Alright, alright, alright !!! I have officially completed three years at Hashnode now. If you have been following my blog for a while, I have this ritual of documenting what all went in the past year at my job.

My general process for documenting is :-

• Go through my commits across different projects and social media interactions.

• Note down high level headings in Apple notes.

• Frame an article out of what I can remember.

You can catchup to all my previous experiences here :-

Blog dashboard revamp

I think this is the largest bit of work that I undertook at Hashnode. Another one that comes close to this is the WYSIWYG editor but yes probably the longest I have worked on the project from start to finish and I am excited to share all the tidbits about it here.

Since I joined Hashnode, I have seen our design language evolve into something great overtime. We have worked on bringing consistency to the whole experience and we are still pushing for it everyday here. The main site has gone through good amount of design changes in my tenure. The major change started with our feeds and then there was an editor design revamp.

But we knew that our blog dashboard felt inconsistent with our new design language. There were so many times we wanted to priortize revamping it but also knew that it’s not a small job and required dedicated engineers to always work on it irrespective of other bugs or features that might be requested.

A high level overview for the work :-

• Creating 23 UI screens from scratch for the whole dashboard in the new codebase.

• Creating new GQL APIs to power this new dashboard

That needs a good amount of PLANNING!!!

The plan

Our usage of Discord forums have increased quite a lot over the last 2 years. For every major feature, we open up a forum and tag in the respective people involved in the decision making. This has worked good so far in practice and we also opened one just called Blog Dashboard for this project.

Our CTO Sandeep shared some interesting insights on why this revamp is necessary for us right now based on conversations with users and the usage analytics. The team also felt aligned with the premise. Our designers already had the first iterations of few screens ready for us based on their convo with the founders.

We use Linear for the project management purposes and man this might be the most PM style breakdown of stuff I might have done in my career till now. Our project was called Rebuilding blog dashboard and the first ticket I created there was to do a small POC to ascertain how to divide the whole project into a bunch of independent tasks that can be picked up.

This POC involved creating the the General page of blog dashboard without adhering to new design for now and integrating it with a temporary UpdatePublication mutation. Doing the POC was helpful in ascertaining the efforts required per screen and also grouping screens together which follow the same pattern.

This exercise resulted in creation of a big Notion doc with projected efforts and doubts which were resolved with help of Jannik Wempe and Sandro Volpicella for all API efforts. The clarifications further helped in creating individual tasks and further bifurcated subtasks for each screen in the blog dashboard. What this meant is that we had figured out close to 60% unknowns and were ready to tackle the remaining 40% as we progressed with the revamp.

We divided the screens in three sets and started with Set-1 of 6 screens. Also everything was featured flag from the start. Only hashnode team members could see the new blog dashboard. Our first set of feedback came from founders and the designers.

The execution

I got to work with Ayodele Samuel Adebayo for the whole revamp of blog dashboard and it was a great collaborative experience. We closely worked on the UI revamp of things. I undertook some of the initial GQL mutations work until Vamsi Rao joined us to speed things up there.

During this work, a lot of components like InputField, TextArea, Select, Breadcumb, Colorpicker and much more were created for our in-house component library. Good amount of work went into translating the design in functioning components that can be reused throughout not just dashboard but also future screens (and it paid off!!).

We completed our first 6 screens of Set-1 before mid of January.

As things were progressing, we realized that the whole revamp will be longer than anticipated and instead of a big-bang release, we will release it in phases on a /beta prefixed route until we can totally replace the existing dashboard. Shad Mirza also joined us to fast track the API efforts here. The feedback from real users is necessary to validate the new experience even if it’s just a set of screens that they can try. So we decided to ship 13 screens in the Phase-1 of the release by Feb end to the end users and boy did we do this with two more days in hand 😎

Okay now I had to take some break off from the blog dashboard work to focus on Docs, specifically the new docs editor and build it along with Rajat Kapoor and Dev Shekhawat . Post that I can jump back to the project again. So before leaving, had a good discussion with team on future milestones of Phase-2. I was only now involved in code reviews of blog dashboard.

I kept a close eye on the blog dashboard progress nevertheless. In Phase-2, the new screens were going on to prod directly to users post a design review and code review. But this also meant that until both reviews are done the thing can’t be shipped to prod. This started to look like a bottleneck to me because sometimes reviews are bound to take time and if hurried just for the sake of shipping to real users, we might compromise quality. So we took inspiration from Phase-1 again and started shipping bits behind feature flag to prod directly. The design review can then happen on prod without the new users getting effected and code review can happen in multiple phases as per the bandwidth.

And this worked well. We have been following this model now for big features at Hashnode.

So after giving almost 2.5 months to the Docs which would be post mid May, I jumped back to blog dashboard and by now we are in the end stage of Phase-2 screens. I worked along with Sam on Domain screen for some final touches and also worked on revamping the Members page Role based Invite system based on user feedback.

We had now 18 screens done and 5 remaining. It was time to gear up for the

We had a good discussion around this again and decided to move old dashboard to a /legacy route and make the /beta one default. Since most of the screens were ready it felt like the right time to make them default for all users and go more aggressive on user feedback. The remaining 5 will get redirected to old dashboard until one by one they land up in the new one. And finally after 2 more months of work and multiple discussions, we finally shipped the remaining screens with the last one being GitHub. And I didn’t plan it but it took exactly 9 months to get to this stage. It’s almost like a baby 👶.

Taking lead on this was a learning in itself. I realized that besides the technical bits of a long project, how effectively you organize and communicate the progress is equally important. We did more than 50 progress update messages in this forum over the course of 9 months. That averages out to at least 1 update/week which kept us going and pushing forward more and more.

Btw, this is what very first push to prod looked like for Blog dashboard :-

This is what it looks like now :-

Docs editor

Now when I took a break from implementation of blog dashboard, I was building stuff on the editor forefront of our new docs offering.

Btw, this is the undisclosed POC that I teased here in my last year end experience post. Though we redid all of the stuff from there in terms of UI and backend, I am glad this took shape in a released product for users.

When I was brought in we were just starting in with editor progress. My role was to figure out everything related to the left sidebar which involved :-

• A guide selector to navigate through diff guides + creating a new guide

• Create, Update, delete and move (drag and drop) all sorts of documentation items (section, page or link).

• Update existing Guide settings

The most interesting bit out of this was the drag and drop experience. I went through different libraries here and experimented with them. We finally went with react-complex-tree. It’s a nice accessible headless tree library needed to build experiences like so. But there was an issue with DND experience where the drag line didn’t appear at the expected position when dragging the items. The reason was that not all items within our UI were equally spaced and had padding or margins around them. The library needed fixed height of each item which I got to know as I dug up existing issues and discussions. So this was an essential problem to fix.

That’s when I started looking into the core code responsible for the logic and pair programmed with Claude 3 sonnet for 2-3 days to figure out a solution where we can patch the library and make our requirement work. And we did it !!!.

I shared the solution on the OSS discussion as well if someone is interested. Though I think the latest versions of the library might have solved it (haven’t gone through that yet). We also wanted to choose an existing solution to not invest a lot of time in building it from scratch but I would now recommend to look at Pragmatic drag and drop if you have good time to build a fully controllable solution. I am yet to actually dig in it but it comes from the creators of react-beautiful-dnd and is a successor to that.

So after building the left sidebar for docs editor, there were couple more stuff that I got involved in from bringing editor as close as to the design possible which involved refactoring some existing implementations. And also integrating the editor with the published view.

I also started getting involved again in blogs dashboard implementation work again after 2.5 months of docs work.

Now I was context switching between blogs and docs since both had their launch priorties.

Docs client

This time the work was on the published view of your docs or what end users will actually see. This was primarily being handled by Rajat and Harsh at the moment. A lot of stuff over here was built in a quick POC fashion so it needed some good amount of refactoring to evolve into a product.

Layouts, layouts and layouts

This might be the 100th time I am working on layout related UI stuff at Hashnode. My first stint was the feed layout and here we are again.

I worked on bringing the header and left sidebar closer to our intended design along with functionality. The sidebar was built from scratch using learnings from the docs editor.

Preview mode

Then I worked on to support preview mode here. So just like your blogs, you want to preview how the thing will look when published, you also want to do same for your docs.

This was done in collaboration with Florian Fuchs. Fun fact, I also worked on supporting the preview mode on blogs during my first year at Hashnode.

A good amount of reusable code was written when supporting this. Building on top of it has made things predictable now.

Performance improvements

We wanted to improve the first interaction experience for our users when they click on a documentation page. To do this we tapped into browser prefetching . I got to learn about Speculation Rules API while working on it. We built a hook that relies on speculation rules if a browser supports it or fallbacks to <link rel=”prefetch”>.

These are the top three that come into my mind when writing this article but a lot of work went on the whole docs product from editor to dashboard to the published client view with close collaboration between different team members. I think we also benefitted a lot from using the same patterns & components that were created during our blog dashboard work to fast-track development of docs features.

Onboarding revamp

Just before launching our docs offering, we also wanted to simplify the experience of creating a blog or documentation. For this, we discussed internally a lot and came up with the final decision of having a new home screen where users will see their existing blog and docs project as well as can create new ones from there. We moved the feed to /community route. This was a big move since most of our users are used to the feed when they open HN.

Simplification also meant that our onboarding experience doesn’t differentiate between a blogs and docs user. Our onboarding was fined tuned for blogs but the time was here to simplify it further and just have one screen to let a user quickly create their account and get started with their first blogs or docs project.

I remember working on Onboarding UI revamp when I joined Hashnode. This was the second time I was touching the bits around here.

Since this was a crucial entry point for new users, we implemented this in a featured flag way so that existing users still see the old onboarding until we were ready to ship both new home page and onboarding together.

Here is how the new onboarding screen looks like :-

Mermaid support for Docs and Blogs

Well the good news is that both our offerings now support mermaid diagrams 🎉.

Actually the first efforts here dated back to our Internal Hackathon last year where I tried to support it for blogs but wasn’t happy with the solution that I came up during the hackathon week. So I discarded it.

This year Sandeep reached out to me if we can do it for Docs. Now Docs has a mdx renderer pipeline which makes creation of custom react components easy and with initial POC efforts the last year + some more digging to make it work with mdx, I was able to validate it that it will work here. So we first shipped Mermaid support for our Docs. Here is how it looks in practice :-

Then few days later, I was reached out again to see if we can make it work for Blogs. I knew I wasn’t happy with the old solution there but having worked on Docs, I got a new idea of tackling this problem and that solution worked well. The editor abstractions are same in both docs and blogs. Only efforts that went for blogs were regarding the render flow in published view.

It’s funny how a product (docs) that was created after blogs got a feature first and some inspiration from work over there helped us bring the same feature to the blogs product.

Quality of life improvements

Graphql Eslint plugin to catch caching issues

As we got into using GraphQL queries and mutations to build features at Hashnode, we ran into a bunch of bugs. The main problem was that the id was missing in the queries and mutation responses. This becomes a real headache when you do a mutation and expect the next query to give you updated results.

Telling developers to always remember to grab the id field wasn't a long-term fix, especially with our fast-paced development and how easy it is to miss things.

So I worked on implementing a eslint plugin which can solve this problem for us and wrote my second engineering article here.

Deploying In-house Avatar generation service

As part of new onboarding, we also shipped in-house avatar generation service. Earlier we relied on ui-avatars.com to generate avatar for each user but wanted to use our own font for the new ones to match with our design. Fazle Rahman worked on this avatar service and I got looped in to bring it to deployable state along with UI integration. Locally, the service needed a puppeteer instance to convert svg to png which wasn’t going to work in a serverless world where the service had to be deployed without some extra deployment work. So I looked for alternatives there and came across sharp first which worked in practice but didn’t allow use of custom fonts which defeated the purpose. Ultimately , resvg-js lib came to the rescue to solve this problem in serverless environment.

Feature flagging + design/code review optimization

I think this was one of the major learning that has worked well for stuff you see being created at Hashnode for more than a year now.

I personally advocate a lot for feature flagging (FF) stuff and getting it to production fast using this approach. This doesn’t mean that we will just ship bare bone to production on a FF. But it means that after the developer themselves have thoroughly tested the feature in their local dev along with ensuring quality metrics, it’s time to get it to production for internal team members.

I have already spoken about it’s advantage in the blog dashboard section but to reiterate:-

• This helps in validating design with production data.

• This enables the code reviewer to take their time to review the code properly.

The quality of product that you can ship to the end user with this approach will be higher because of the confidence you have with how it looked in production when the internal team tested it. Again, our scale of team and product is different so experiment with what works best for your team and product.

Hacking a side project that converted articles to songs

I was able to hack a side project called Hasunode over one of the weekends this year which would convert a Hashnode article to a song 🎶. And it was well received by the users. In fact, it actually won the first place in one of the Peerlist project weeks ✨.

Hashnode workation in Bangalore

Almost our whole team assembled in Bangalore, India to work together under the same roof for a week during the month of October. We also created a one week of roadmap items that we want to target only in this week. This included stuff other than our ongoing feature work. It could be any quality of life improvement, a small feature etc which you can wrap up in a week. This is when I actually shipped the Mermaid support on Blogs.

We also had some interesting conversations regarding our future roadmap and how things are looking for us right now.

We celebrated Jannik’s birthday together and had great food and drinks at different places. I also got to play Badminton with the German folks and had intense matches with them. Got to meet and have interesting talks with our ex-shippers at Hashnode - Girish Patil and Rajat.

Also, I was going to complete my 3 years in the next week of the workation so the team surprised me with the early celebration with my favourite Lotus biscoff cheescake 😋

We worked out of this Coworks in Bellandur for a week 👇

Running into Ashish Vidyarthi at our Hotel was not something I had in my list but great to interact with him for those few seconds

Some of us went to Cubbon park in early morning of our last day. Jogged 2km along with Jannik to be out of breath and ending up having a 160 bpm heartbeat while him being at 130 bpm and me questioning all of my life choices. But it’s super fun to be competitive with him.

Also on the last day, enjoyed spending time with the team doing Go-karting, Laser tag and Bowling. Overall it was a great experience from stay to all goodies we got 😉 and obviously getting much more familiar with in person thoughts and culture of the team members.

What’s cooking now ?

Well as some of you might know that post our launch of Docs by Hashnode, we have been actively listening to customer feedback and improving the offering. Since then a lot of things have been shipped already and one the major feature that cater to our Startup and Enterprise plan users is going to be released very soon. It’s bringing versioning support to the docs.

This again required some good amount of discussions and planning to execute to ensure everything works well for existing users while we build this in a iterative fashion.

In fact, we have been shipping it incrementally over feature flags and it’s almost there. I got to collaborate with Kieran Roberts and Jannik for building this.

Excited to bring it live for all users !

Here is a sneak peak of the editor and published view :-

What else ?

Well this is a long article already than what I had in mind. If I go on rambling about some interesting bug fixes and more things that I got to ship, it’s gonna take more words and probably bore out the readers. Also I have realized that I can actually share this kind of stuff in set of posts on X or Linkedin. Here is a list of misc interesting bits I shared over social media which touches aspects of my work at Hashnode :-

Wrapping up

Well it’s been another interesting year at Hashnode. Besides the technical bits, I got to interact with almost all team members to work on one or the other aspects of the products. This probably was the year of collaboration for me. Also leading one of the longest running projects of my career till now in a startup is something great to think about personally. We are collectively figuring out stuff at Hashnode on a daily basis but I am proud of all that we have done till now as a team 🚀.

Well here we are reflecting on my time at Hashnode the previous year. I have officially completed two years at Hashnode and as a part of ongoing ritual where I document the experience, here is another one.
And also, in first quarter of 2023, I got promoted to a Senior Software Engineer so it was a great year from career prospects.

Here are the previous reads for those interested :-

Lets straight away dive into what all was I involved in the last year.

Blog UX improvements

This began in the end of year 2022 around first week of December. If you have been using hashnode as your blog provider for a while, you might have noticed that the design an year ago vs how it looks right now is different. Our designers worked on an overhaul for the same.
I got to collaborate with Kieran Roberts , Girish Patil and Vamsi Rao on these. We do things now in a much better way than we did for this one honestly (in terms of the DX). It was a set of huge PRs that was reviewed and merged in early weeks of 2023.
I specifically got to work on the comments sidebar and bits of the floating toolbar for each article. In the floating toolbar, the logic empowering how the click on the heart works and hits our backend API in an optimistic way was something I enjoyed building.

Also there were not only UI facing changes but also significant database modeling and re-syncing changes. We almost got everything right. But the Friday deployment curse did catch up to us.

Since there were a lot of changes that were done in a short period of time, the following weekend, I spent time on evaluating if everything is functioning smoothly. Thats when I encountered that we missed syncing one important field to our db model which resulted in a reset of a particular model field for all users. Luckily, we patched it the next day by running a script which took care of merging the new field values with the old one and restoring data to consistent state.

Blog performance improvements

After the UX overhaul, I got to spend some time in assessing areas where we can improve performance aspects of our blog. This included lazy loading resources which do not need to be shown above the fold, primarily the images and embeds + dynamically loading components which are needed on interaction and some bits about optimizing LCP elements. We actually did a twitter space where around the 18:50 mark, we start sharing the updates for the performance.

Undisclosed POC

This was a new project from ground up. I collaborated with Sai Krishna Prasad Kandula on this for almost 2.5 months. This included white boarding db models, flows, UI and then bringing them to life and keeping them accessible to our internal team for feedback. But whats the future for it ? Its something which is being now proactively evaluated by us and might even be in the upcoming roadmap 🤞.

Hashnode Pro

I got to work on the Focus mode in our editor which is a part of our Hashnode Pro offering. Collaborated with Joshua 'Navi on this one and possibly our first collab 👊. I became a fan of zustand when we worked on this since we introduced that while building this feature.

Also contributed in building the landing pages for hashnode.com/ai and hashnode.com/pro. The one section that took most time for me was the AI section of the hashnode.com/pro page . Enter framer motion 🤌.

Login/logout patches for hashnode blogs

I swear I needed cookies at this time to solve cookie related issues. Again got to collaborate with Girish Patil on this one. This was solved in 2 phases and since then we haven't had any specific complains where a user login/logout on their blogs was not working as expected. The initial fix was strictly applied for me and Girish and later on for everyone.

NextJS middleware is honestly a powerful thing. I think after enough discussion and what ifs , we decided that middleware is the best place for these fixes to exist. We used the req.cookies API at the time to solve this. But their cookies API at that time was something which worked very differently than if you are used to the `req.headers.set` approach. Later on I think we ditched that API and just used headers API since it worked as expected and had lesser unknowns.

Subscriber Modal

As part of our newsletter feature, we allow users of Hashnode to enable a subscriber modal to be shown at the end of each of their article's. This was primarily built by Kieran Roberts. There were two parts to this feature. Showing up the modal and taking action when the user interacts with the modal. It's the second part where I got to contribute along with Kieran. When the user lands up back on the article after creating their account on Hashnode, there were a set of checks that were needed to be done to let the user know about what exactly happened and what they can now expect. This was communicated via different toasts based on internals states.

Meeting the folks at Bangalore

Got to meet the folks during mid of the year at Bangalore and work alongside them ✨. This is always a refreshing thing to do. We also did a nearby morning trek together ⛰️.

Kicking off component library

This was primarily initiated and implemented by Rajat Kapoor and the designers - Pritam Paul and Joshua 'Navi . I got to contribute on things like handling how the local development can work with this library without publishing this as a package. As of now, We have been following the local + publishing pre-patches flow for some while to do rapid development. We still have to make efforts to make it consumable throughout Hashnode. But I am glad efforts were started in this direction.

RHPC - Reduce Home Page Clutter

Yes quite a code word there. This was the first project that was done after kicking off the component library so most of the components got made during this project.

This is where we restructured our feed layouts from 3 to 2 column and also redesigned the feed UX + right sidebar + notifications and profile menu.

However, the project's execution didn't go as smoothly we expected it to go. There were some user facing quality issues at the end (which we quickly patched) but nevertheless I specifically had two learnings/realizations post this :-

• Things were not streamlined enough for everyone to know where we can make that tradeoff of quality and speed.

• Having more amount of engineers involved in the same project doesn't necessarily mean more output. Sometimes less is more and identifying these early on is better.

We also had a Engineering Alignment call and also published a Product Coding Guidelines as part of our PR template to go through before raising a PR for review. It was something I made as notes for myself while working on different features and bugs at hashnode and have been wanting to make it accessible to our engineering members.

Discussions

Taking all the learnings from the previous project, we got into developing Discussions. We wanted our users to explore what other users think about the articles on our feed and indulge in healthy discussions for the same. So we made it easier for the users to post the comments straight from hashnode.com via this feature.

This is my favorite piece of work in 2023. We were proactive in pushing things behind feature flags to prod from day one. I got to work on refactoring our feed cards to be able to handle new discussion related CTAs and links. Also got to work on new Commenting system UI implementation + integrating our new GQL APIs with it.

Headless Hashnode

Headless Hashnode is our newest offering from hashnode and a really powerful one. It allows you to totally own the frontend of your blog and start creating them on your own subpaths (for eg /blog).

I think most of my tinkering around here happened to improve the lighthouse scores for our starter-kit around performance, accessibility and SEO. This was purely pouring in the experience earned while working on the blogs offering and making sure that our starter kit also benefit from the same.

Social media marketting

Okay thats a made up category for a Software engineer but hey I did create two videos to promote Hashnode features. The first one was for focus mode

The second one was for headless hashnode and the anime lovers (JJK fans) are going to love it .

It feels nice to have this sense of freedom to be creative and promote the work done by your company in your own way.

Hashnode Fitness challenge

Well, as of writing this, I am not as fit as I was when I secured the 3rd place in the September fitness challenge for Hashnode. I will start the routine soon. For me, this included running 5kms every morning and I achieved probably my peak fitness of 2023. Oh how the winters make me super lazy (smartly blaming winters).

Quick wins

So we also introduced a notion of quick wins and sharing them with others. I think as part of this I have few of them to highlight :-

• Solve users' problems directly via social media / GitHub - At Hashnode, we often reach out to users on Twitter or other platforms when they encounter issues and help them resolve their problems.

  

• Added a perf related improvement to cut the loading time of our /users api by 300ms . Example of small change -> big impact.

• Our Vercel builds were constantly failing due to massive size of our assets. But solving that would have taken more engineering efforts. And the builds were running for like 20 mins only to fail later which resulted like in like at least 30 mins of daily time being wasted just to deploy our changes. So, I went through the official troubleshooting docs here and some GitHub issues and found that this was prominent if you had BUILD_CACHE being used (the default setting). So we simply turned it off and now our builds don't fail and take an average of 10-15mins, a tradeoff, we were happy with.

• Introducing Dark mode toggle straight from the editor. As much as it became easy for a user to switch themes, it also became easy for the devs to quickly test theme related changes in the editor. A win-win for both the parties.

• We removed the Write button from our dashboard header some time back as it was not optimized to work for different blogs but the users gave us feedback that it should be there. So I got to work on re-introducing it + adding the capability to retain the context of the blog for which the button was clicked so that a user doesn't land up on the editor and needs to reselect the blog. The article is automatically created for them based on from which blog dashboard they clicked the button.

Hashnode Internal Hackathon

We also had our first internal hackathon at Hashnode where all our team members went onto develop and ship really cool things. My favorites are the Webhooks and the Cmd+K features which were also winners of this hackathon.
I personally got to work on three things out of which two made to production.

• One of them was the EditorV2 overhaul in terms of design and accessibility.

• Another one was font-optimisation for blogs which I covered in this tweet

Weird bugs

I have already shared some stuff in the following article which focuses on debugging issues at Hashnode. These were mostly dealt with in 2022.

In 2023, I have not yet compiled a list of what bug fixes I got to work on except one for which I wrote a twitter thread. This one was UI specific and needed us to a have a zoomed out perspective. I mean literally, I could only replicate it for zoom of 90% on browsers and that helped us narrow down the cause.

There have been some recent ones which I will probably keep for future articles.

Code reviews

I posted a tweet that compared how my GitHub activity looks over my tenure here

One can see that the amount of time spent in reviews has increased for me. Its now balanced with what I work on. For this year, it spanned across features, bug fixes, migrations , refactors etc. Some highlights can be

• Table of contents (done by Ayodele Samuel Adebayo )

• Migrating from twin.macro to simple tailwind setup for blogs (done by Jannik Wempe ) - My biggest code review yet. I took my good time for this.

• Discussions, Blog UX improvements and Security Patches

• Accessibility improvements (done by Kieran Roberts )

And just more miscellaneous stuff I can't remember as of now. And this is not just something I have to do as a reviewer. It's a shared responsibility of all our engineering team members. Over time, I feel as a team, we have become open to reviewing more code and building context around problems. This is necessary to keep unblocking people and ship stuff out there.

Also I wrote another article on how having a broader perspective during code reviews can help sometimes :-

What's cooking ?

Well I have been working on another migration now with an approach similar to how we built Discussions and besides the UI work, it also involves creating new GQL APIs which I have been wanting to do for some time. It's going well as of now and I hope our users will enjoy the revamped experience ✨.

Besides this, I am thankful for a wonderful year I had at Hashnode and being able to collaborate with amazing folks here 💯.

One of the things when using the NextJS ecosystem that hasn't yet clicked for me is their Script component.

I have been running some experiments on how I can leverage it for the Google Analytics script like gtag.js.

Experiment - The Setup

There are two types of deployment involved here:-

• Gtag-Standard - This one uses the _document.tsx file where we are loading the standard script tags with the async attribute as recommended by Google.

    import { Html, Head, Main, NextScript } from "next/document";
  
    const Document = () => {
      return (
        <Html>
          <Head>
            {/* Global site tag (gtag.js) - Google Analytic */}
            <script
              async
              src="https://www.googletagmanager.com/gtag/js?id=G-2VSQZMZCY6"
            ></script>
            <script
              dangerouslySetInnerHTML={{
                __html: `window.dataLayer = window.dataLayer || []; function gtag()
                       {dataLayer.push(arguments)}
                       gtag('js', new Date()); gtag('config', 'G-2VSQZMZCY6');`,
              }}
            ></script>
          </Head>
          <body>
            <Main />
            <NextScript />
          </body>
        </Html>
      );
    };
    export default Document;
  

• Gtag-NextJS - This one uses the index.tsx file where we are loading the Script tags with the afterInteractive strategy.

    import Script from "next/script";
  
    const Home = () => {
      return (
        <main className="h-screen flex items-center justify-center">
          <h1>Google Analytics Test</h1>
          <>
            <Script
              id="google-analytics-init"
              dangerouslySetInnerHTML={{
                __html: `
              window.dataLayer = window.dataLayer || []; function gtag()
              {dataLayer.push(arguments)}
              gtag('js', new Date()); gtag('config', 'G-2VSQZMZCY6');`,
              }}
            />
            <Script
              id="google-analytics"
              src={`https://www.googletagmanager.com/gtag/js?id=G-2VSQZMZCY6`}
            />
          </>
        </main>
      );
    };
  
    export default Home;
  

  Note - Both of the deployments use the pages router setup.

  I referred to the following two resources on how to load the gtag script.

  • NextJS Docs

    

  • next/third-parties - Google Tag Manager

    

Experiment - The Execution

I ran both of the deployments through the webpagetest's Visual comparison test.

This was done 10 times so that we get a good enough sample space.

First Run

Second Run

Third Run

Fourth Run

Fifth Run

Sixth Run

Seventh Run

Eight Run

Ninth Run

Tenth Run

Observations

From all the above runs, there were two interesting bits that I want to highlight specifically:-

• CPU Busy Time - This has been consistently lower for the Gtag-Standard deployment.

• Total Blocking Time (TBT) - This has been frequently lower for Gtag-Standard deployment but maybe with more runs this metric averages out to be the same for both the deployments.

Resources

Deployment Urls

• Gtag-NextJS - https://gtag-nextjs-script.vercel.app

• Gtag-Standard - https://gtag-standard-script.vercel.app

Webpagetest runs

• https://www.webpagetest.org/video/compare.php?tests=231101_BiDcS6_7AR,231101_BiDc6Q_7AV

• https://www.webpagetest.org/video/compare.php?tests=231101_AiDc17_7QY,231101_BiDc4K_7SX

• https://www.webpagetest.org/video/compare.php?tests=231101_AiDcSM_7YZ,231101_BiDcMY_813

• https://www.webpagetest.org/video/compare.php?tests=231101_BiDcP6_81F,231101_BiDcJ0_81G

• https://www.webpagetest.org/video/compare.php?tests=231101_BiDcQ9_821,231101_BiDcNP_822

• https://www.webpagetest.org/video/compare.php?tests=231101_AiDcTG_80M,231101_AiDcVS_80P

• https://www.webpagetest.org/video/compare.php?tests=231101_BiDcXA_82N,231101_BiDcCV_82P

• https://www.webpagetest.org/video/compare.php?tests=231101_BiDcH0_830,231101_AiDcPV_818

• https://www.webpagetest.org/video/compare.php?tests=231101_AiDc2E_81E,231101_AiDcMT_81F

• https://www.webpagetest.org/video/compare.php?tests=231101_AiDcHT_81S,231101_BiDc8G_83C

Conclusion

The whole point of this experiment was to evaluate if using NextJS Script tags, we would get better performance scores when loading Google Analytics JavaScript. But that didn't happen. More or less they are on par with the standard ones. The standard script tag delays the load event and the NextJS Script tag starts after the load event. In either case, I believe, Script Evaluation (of gtag.js) contributes to the TBT metric. I thought maybe loading the gtag script after the browser load event might result in lower TBT but that's not how it works.

Again, I am curious about the best way to load such analytics scripts. They are essential for users but contribute to poor scores for Mobile performance audits. Feel free to let me know if there are incorrect assumptions or gaps in how the above experiment was done.

Thank you for your time :)

This article will touch upon scenarios that I have encountered while working at Hashnode where some code reviews helped us prevent future rework on the same issue by having a broader perspective.

Code Review 0

The Problem

Sometime back, one of our engineers, Kieran Roberts , stumbled upon a performance degradation when he was working on upgrading the navigation header behaviour for the blogs.

The issue wasn't strictly related to the header but the left sidebar. Whenever we closed the left sidebar on the blogs, the CPU spiked up to 100%.

This behaviour happened only in Chrome. Now that spike also affected our new header UX because it relied on doing some fancy listening to scroll events and style updates. The header navigation became janky. So this was a blocker that we wanted to get rid of first before shipping the header update.

As far as I remember, I think he found a workaround that prevented the CPU spike but did require making a tradeoff at that time which I can't exactly remember. When the PR for the same came to me for code review, I started digging into the GitHub issues around this behaviour in radix (a set of powerful UI primitives that we have been using lately to power a lot of UI on Hashnode) and ran into this issue thread:-

That is where I found out that the maintainer patched the behaviour a day ago that was making their Dialog radix-primitive to cause this CPU spike in Chrome and the patch was available in a release candidate for the same.

The Result

We updated our package version to the release candidate after confirming the same with the maintainer on that same thread and it did stop the CPU spikes.

We were honestly lucky to observe this issue at the right moment. The patch from the OSS was shipped at the right time for us because we ran into the issue just 2 days before it. For the most part, what this meant as a learning experience for me is that it helps to drop by the OSS repositories of the libraries/frameworks being used in your work to check if the bug/regression exists at the implementation end of that OSS code and not where it's used in your project.

Code Review 1

The Problem

This is much more recent, I think within the last month in fact, where a minor vulnerability was reported by a user.

I was again tasked with doing a code review for the PR which addressed a part of that vulnerability. Mostly, the PR dealt with addressing it more at UI end by modifying some bits. We were still unsure of where exactly at the backend API, one or the other check was missing which might have caused it and wanted to first ensure that at least from the UI end, it's not doable anymore. Post that, we would have investigated further.

As I was navigating through the PR, I thought just once to check the API implementation which existed in a separate repo. I found out that we already have the required checks in place and there was no reason that this vulnerability should exist. At that moment, I looked up when these checks landed in the code and it was again

Talk about coincidences.

The fix happened at the API level as part of a refactoring of one of our GraphQL mutations by Florian Fuchs . He was not aware of this issue and we were not aware that this was fixed just a day ago. But again, lucky sighting. To be double sure, we looped him in and got confirmation that indeed this was fixed by the refactor.

The Result

The confirmation closed the loop there itself. The timing was critical here because when Kieran could replicate the vulnerability, he made a UI-level patch as per it but when I code-reviewed it, the patch had already been done at the API level so there was this gap which needed to be bridged. What worked here well was that during the code review, just out of curiosity, I checked the API repo and didn't restrict myself to the scope of the original review.

Conclusion

It's weird that both these code reviews involved me and Kieran and had that one-day fix factor. I think these cases might be rare but since they have happened twice with me already, I thought of sharing these to maybe open up another perspective for other developers who are in the middle of making or reviewing that PR and can use it.

Thank you for your time. Do you have such experiences with code reviews? Feel free to drop a comment and share them. You might just save another developer a good amount of time.

You're scrolling through Twitter (I still like calling it that) or Medium and you get a nice view of the sidebar as it snaps to the bottom or top of the page if you try to scroll beyond its boundaries.

We will re-create the same experience through a react implementation.

But before that, let's observe how the properties of these sidebar elements mutate as you scroll the viewport in existing implementations.

Observations

• Twitter - On Twitter, as you scroll from top to bottom of the screen and the bottom of the sidebar comes into view, the sidebar's position becomes sticky with a mutated top value. There is also a margin-top on the sidebar element. Now if you scroll from bottom to top the sidebar remains sticky but now with a mutated bottom value. Again, the margin-top value changes in this case.

• Medium - On Medium, as you scroll from top to bottom of the screen and the bottom of the sidebar comes into view, the sidebar's position becomes sticky with a mutated top value. Now if you scroll from bottom to top, the sidebar's position is relative along with a mutated top value.

I found that Medium was able to get the bottom stickiness working without explicitly setting a bottom value and no margin-top manipulations (Although margin-top does exist on the element, removing it didn't do anything). So only two properties, namely top and the position needed to be mutated if we went with this implementation.

Considerations

With that in mind, I first thought of an Intersection Observer implementation where I will use two pseudo divs wrapping the actual content and observing their intersection with the viewport to make certain decisions on how to mutate the sidebar properties.

Here is how the HTML structure for this implementation looked like:-

<div> - Sidebar element
    <div/>  - Top pseudo div
        {children} - sidebar content
    <div/> Bottom pseudo div
</div>

While making this work, I realized that it might be hard to not use stuff like getBoundingClientRect or offsetHeight properties to compute certain cases which Intersection Observer (IO) might not account for. I thought to avoid those for performance reasons as reading them may result in reflows whereas reading values inIO doesn't cause that.

So I shifted to pure DOM calculations from the IO approach to keep it standardized. It might not be the most ideal approach but works well.

Here is how the HTML structure for this implementation looked like:-

<div> - Sidebar element
  {children}
</div>

Again, the calculations need to happen inside a scroll listener because the logic depends on whether you're scrolling up or scrolling down.

Implementation

The full-fledged implementation is available below but I will be sharing and explaining those bits which cover the conditions that govern when the sidebar is sticky and when it's not.

Let's observe some set of important conditions(inside a scroll handler) that give the sidebar that sticky behaviour

if (isScrollingUp) {
  if (isTopEndBetweenViewport) {
    setPosition('sticky');
    setTop(topPositionAtWhichElementBecomesStickyFromTop);
  } else if (isBottomEndBetweenViewport) {
    setPosition('relative');
    setTop(topDistanceOfElementRelativeToPageTop - offset);
  }
} else if (isScrollingDown) {
  if (isTopEndBetweenViewport) {
    setPosition('relative');
    setTop(topDistanceOfElementRelativeToPageTop - offset);
  } else if (isBottomEndBetweenViewport) {
    setPosition('sticky');
    setTop(topDistanceAtWhichElementBecomesStickyFromBottom);
  }
}

Explanation:-

• When a user is scrolling up:-

  • If the top of the sidebar is visible, then we set it's position as sticky. We also set it's top value to what we want it should start sticking from.

  • If the bottom of the sidebar is visible, then we set it's position to relative. We also set it's top value to what we know the element itself has offset by from the top of the page. So suppose you're at 1234px from the top of the page then top will become 1234px.

• When a user is scrolling down:-

  • If the top of the sidebar is visible, then we set it's position as relative. We also set it's top value to what we know the element itself has offset from the top of the page. So suppose you're at 436px from the top of the page then top will become 436px.

  • If the bottom of the sidebar is visible, then we set it's position to sticky. We also set it's top value to what we want it should start sticking from the bottom. This will be mostly a negative value.

Now let's see more conditions within the scroll handler that do not depend on the scrolling direction but instead are required to exit early even before reaching the direction-based logic we discussed above:-

if (isTopEndBelowViewport || isBottomEndAboveViewport) {
  setPosition('relative');
  setTop(scrollYOffset);
  prevScrollTop = scrollYOffset;
  return;
}
if (areBothTopAndBottomEndsOnOppositeEndsOfViewport) {
  setPosition('relative');
  setTop(topDistanceOfElementRelativeToPageTop - offset);
  prevScrollTop = scrollYOffset;
  return;
}

if (areBothTopAndBottomEndsBetweenViewport) {
  setPosition('sticky');
  setTop(topPositionAtWhichElementBecomesStickyFromTop);
  prevScrollTop = scrollYOffset;
  return;
}

Explanation:-

• If the top end of the sidebar is below the viewport or the bottom end of the sidebar is above the viewport, we will set it's position to relative and its top to scrollYOffset (alias for window.scrollY). But when can we run into such a situation? It's possible while navigating between pages that the sidebar unmounts and mounts or some race conditions caused it to not be at its correct position. More of an extra check that can help mitigate those.

• If the top end of the sidebar is above the viewport and the bottom end of the sidebar is below the viewport, we set its top value to what we know the element itself has offset by from the top of the page.

• If the top end of the sidebar and bottom end of the sidebar are both visible in the viewport, we set its position as sticky. We also set it's top value to what we want it should start sticking from. If the sidebar is too long, this becomes more apparent in the zoomed-out view.

Let's explore the variables used in the above code snippets:-

• topPositionAtWhichElementBecomesStickyFromTop - This is user-provided top value. By default this is 0. You might want to change it to something else as per your needs. The sidebar calculations will factor in this property.

• topDistanceOfElementRelativeToPageTop - This is calculated using the element.offsetTop property. offsetTop is calculated relative to its nearest positioned parent, also called the offsetParent. If no such positioned parent is present, body is used for calculation.

• offset - This is also the element.offsetTop property but the one that was calculated initially outside the scroll handler. This is used to account for stuff like padding, margin, position etc. and is needed to factor in the upcoming scroll handler calculations.

• topDistanceAtWhichElementBecomesStickyFromBottom - This is the element.getBoundingClientRect().top value. getBoundingClientRect() works by giving values of the element's position relative to the viewport. So when the bottom of the sidebar is visible, the getBoundingClientRect().top will generally result in negative values for the top as it would be above the current viewport in most cases. So we can set that as the top value along with position as sticky to snap the sidebar to the bottom of the viewport.

All of the above code logic is abstracted in a useStickyElement hook. This hook is then consumed by a Sticky component which looks like below:-

export const Sticky = React.forwardRef<HTMLDivElement, ComponentProps>((props, ref) => {
  const { children, top, as = 'div', ...restOfTheProps } = props;
  const Element = as;
  const stickyContainerRef = useRef<HTMLDivElement>(null);
  const mergedRefs = composeRefs(ref, stickyContainerRef);
  const { top: _top, position } = useStickyElement({
    elRef: stickyContainerRef,
    top,
  });
  return (
    <Element
      ref={mergedRefs}
      style={{
        top: _top,
        position,
        height: 'fit-content',
      }}
      {...restOfTheProps}
    >
      {children}
    </Element>
  );
});

In the above code, the height: 'fit-content' CSS property is important. This is what allows the element to not occupy more space than the content it has and the hook calculations to work. If it's not present, the sidebar's height will fall back to auto which will make it occupy the rest of the unused space.

Limitations

The current implementation works great for cases where the elements above the sidebar won't resize or disappear. But if they do, the implementation cannot account for those because it relies on the initial element.offsetTop. There is a way to make that work by simply umounting and mounting the Sticky component to account for DOM changes and reinitialize the calculations. We can use different keys on the Sticky component in such cases that automatically ensures it.

Playground

Check out the GitHub repo where you can see the full implementation and also access an example to play with the sidebar by dynamically setting the elements on the main content or the sidebar itself. I have intentionally skipped sharing the verbose bits of the useStickyElement hook in this article to focus on more conditional logic which makes it work.

Conclusion

The implementation doesn't try to be generic but is specific to the use case of getting that Sticky scrollable element behavior as we see in the sidebars.

For the readers of this article, I have some open-ended questions for you:-

• How would you have implemented it differently? Do you have a less JS and more CSS-based implementation in mind?

• What optimizations can be done to the current implementation?

• What libraries are doing this great? For eg; I played with react-stickynode and react-sticky-box before writing this implementation. They both worked fine but I noticed with dynamic loading content (infinite scrolling) or zooming in and out, there was some jumpy behavior. Although they are made with much more configurability in mind.

In this article, I am going to be elaborating on how I used GPT-4 to write python code to detect Dysarthria speech and output the transcribed words. But basically, I couldn't get it to work due to a lack of domain knowledge.

Introduction

Back in 2017-2018 during my college years, I worked on a project with my friend about improving the intelligibility of dysarthric speech. Let's define some of this jargon for you:-

• Dysarthria - It is a motor speech disorder that affects the ability of an individual to articulate words properly due to weakness or poor coordination of the muscles used for speech. It can be caused by various medical conditions, such as neurological disorders, brain injury, or degenerative diseases.

• Intelligibility - Intelligibility refers to the degree to which speech can be understood by others. In the context of dysarthria, it refers to the degree to which individuals with dysarthria can convey their intended message through speech.

So all in all, we wanted a means to improve the understanding of words spoken by dysarthric people. At that time, Data Science & ML were very interesting things to me even though I lacked domain knowledge. We tried solving it through the means using some RNNs and stuff like that but I will be honest, we couldn't even get close. The reason was simple. We lacked domain knowledge. We didn't have the mentorship of a domain expertise person. All we did was google stuff, trying to play with hyperparameters around the data we gathered, cleaned and prepared for training.

So we abandoned the whole ML approach and couldn't proceed further on it.

At that time, I was all into python code. I knew python the best as far as I can remember. This is not true anymore. I have been writing JS professionally now for almost 3.5 years.

But now we have LLMs filling in the lines of code for us in languages we might be less aware of.

With the per-day advancement that's been going on, I wanted to revisit the college project from a different angle this time. Instead of trying to improve intelligibility, I wanted to know if we can build a POC ASR (Automatic Speech Recognition) tool for dysarthric people to transcribe the words spoken.

I told this same problem statement to GPT-4 and it did spin up some python code for me. Again, I still lack the domain knowledge of the whole ML space but I still wanted to give this a try to see where I will hit the wall finally.

In this article, I am going to utilize Hashnode's AI tools to summarise and explain the code that was written by GPT-4. It's only legit if an AI explains the work of another AI. Think of me as an instructor while the code and explanation are being done by the AI toolset.

The Dataset

It goes without saying but data is of the utmost importance when it comes to trying to build a corpus to train on any model.

I have the data obtained from UASpeech Database. This consisted of audio recordings of basically two types of speakers:-

• Dysarthric: The speakers are categorized as F02, F03 (for female speakers), and M02, M03 (for male speakers), with each index representing a unique speaker.

• Control: The speakers are categorized as CF02 and CF03 for females and CM02 and CM03 for males, respectively.

So, to figure out what's going on with dysarthria, scientists have recorded a bunch of audio of people with it talking. But, to make sure they're getting accurate results, they need a control audio too.

This allows for a comparison between dysarthric speech and normal speech, helping scientists identify the specific areas of speech that are affected by dysarthria.

For each dysarthric audio, there is a corresponding control audio. For example, if a dysarthric individual speaks the word "Command" in a labelled file named F02_B1_C12_M2.wav, then there is a corresponding control audio named CF02_B1_C12_M2.wav with the same utterance of "Command".

Hey, just a heads up - right now there aren't any written versions of these audio files. So basically, we can't see exactly what's being said. But we will get around it by generating the same via GPT-4's help.

Cleaning up the dysarthric audio

Our focus is on solving the challenge of understanding dysarthric audio. The issue lies in the prolonged duration of individual words and the presence of empty audio gaps. For now, we are not concerned with dialect or pronunciation. Furthermore, background noise amplifies the difficulty of predicting the intended word. To address these concerns, we tasked GPT-4 to provide us with Python code that performs noise reduction and voice activity detection.

import librosa
import numpy as np
import webrtcvad
import os
import torch
import torchaudio

import noisereduce as nr

def noise_reduction(waveform, sample_rate):
    y = np.array(waveform)
    y_reduced = nr.reduce_noise(y=y, sr=sample_rate)
    return torch.Tensor(y_reduced)

def voice_activity_detection(waveform, sample_rate, vad_window=30):
    vad = webrtcvad.Vad(3)
    frame_duration = vad_window / 1000.0
    frame_length = round(sample_rate * frame_duration)
    num_frames = len(waveform) // frame_length

    # Convert waveform to int16 and scale it
    waveform_int16 = (waveform.numpy() * 32767).astype(np.int16)

    active_frames = []
    for i in range(num_frames):
        frame = waveform_int16[i * frame_length : (i + 1) * frame_length]
        is_speech = vad.is_speech(frame.tobytes(), sample_rate)
        if is_speech:
            active_frames.append(torch.tensor(frame, dtype=torch.float32) / 32767)

    if len(active_frames) > 0:
        return torch.cat(active_frames)
    else:
        print("No speech frames detected.")
        return torch.tensor([], dtype=torch.float32)


def load_audio_librosa(file_path, sample_rate=16000):
    waveform, _ = librosa.load(file_path, sr=sample_rate)
    return torch.from_numpy(waveform), sample_rate


output_folder_vad = "/path/to/audio/files"
os.makedirs(output_folder_vad, exist_ok=True)
# load from Audio/F02
input_audio_files = librosa.util.find_files("/path/to/audio/files", ext=["wav"])

for input_audio_file in input_audio_files:
    try:
        waveform, sample_rate = load_audio_librosa(input_audio_file)
    except:
        print("Error loading file: {}".format(input_audio_file))
        continue

    # Apply noise reduction
    waveform_denoised = noise_reduction(waveform, sample_rate)

   # Apply voice activity detection
    waveform_vad = voice_activity_detection(waveform_denoised, sample_rate)

    # Save VAD processed audio if the tensor is not empty
    if waveform_vad.numel() > 0:
        output_vad = os.path.join(output_folder_vad, os.path.basename(input_audio_file))
        torchaudio.save(output_vad, waveform_vad.float().unsqueeze(0), sample_rate)
    else:
        print(f"Skipping {input_audio_file} due to empty waveform after VAD.")

Explanation: The above code performs noise reduction and voice activity detection (VAD) on audio files. It uses the librosa library to load audio files in WAV format, applies noise reduction using the noisereduce library, and applies VAD using the webrtcvad library to detect speech segments in the audio. The active speech segments are then saved to a new file in the specified output folder if they are not empty. The VAD algorithm used here is based on the WebRTC implementation and the vad_window parameter specifies the time (in milliseconds) for each VAD window. The code also uses the torchaudio library to save the processed audio files.

Upon initial creation, the code did not meet expectations as the custom noise reduction logic produced subpar audio quality compared to my loaded audio. After 3-4 attempts of instructing GPT-4, I was able to generate the above code which provided improved audio quality surpassing my loaded audio. I thought the result was good enough for training.

Transcribing the control audio

As previously mentioned, each dysarthric audio has a corresponding control audio, which can be utilized to obtain transcriptions. Therefore, I instructed GPT-4 to use the control audio for this purpose:-

import os
import pandas as pd
import speech_recognition as sr
import time

def transcribe_wav(file_path, retries=3, timeout=10):
    recognizer = sr.Recognizer()
    with sr.AudioFile(file_path) as source:
        audio = recognizer.record(source)

    for i in range(retries):
        try:
            return recognizer.recognize_google(audio, language='en-US', show_all=False)
        except sr.UnknownValueError:
            print(f"Could not understand audio in {file_path}")
            return ""
        except sr.RequestError as e:
            if i < retries - 1:
                print(f"Request failed, retrying ({i + 1}/{retries})")
                time.sleep(2 ** i)
            else:
                print(f"Could not request results from Google Speech Recognition service; {e}")
                return ""

def save_progress(data, output_file="transcriptions.csv"):
    header = not os.path.exists(output_file)
    df = pd.DataFrame(data, columns=["file_name", "transcription"])
    with open(output_file, 'a') as f:
        df.to_csv(f, index=False, header=header)

def load_progress(input_file="transcriptions.csv"):
    if os.path.exists(input_file):
        df = pd.read_csv(input_file)
        return df["file_name"].tolist()
    else:
        return []

control_folder = "control_train"
output_file = "transcriptions.csv"

output_data = []
processed_files = load_progress(output_file)

for file in os.listdir(control_folder):
    fileWithoutExtension = os.path.splitext(file)[0]
    if file.endswith(".wav") and fileWithoutExtension not in processed_files:
        file_path = os.path.join(control_folder, file)

        transcription = transcribe_wav(file_path)

        if transcription == "":
            continue

        # remove .wav from file name
        file = os.path.splitext(file)[0]

        output_data.append((file, transcription))
        save_progress(output_data, output_file)
        print(f"Saved transcription for {file}")

Explanation: The code is transcribing audio files in WAV format in the control_train folder using the Google Speech Recognition service. It first defines a function called transcribe_wav takes in the file path of an audio file and uses the SpeechRecognition library to recognize the speech in the audio file using the Google Speech Recognition service. If the service fails to recognize the speech, the function retries up to three times before giving up. It then defines two helper functions called save_progress and load_progress that respectively saves and loads the progress of the transcription process to and from a CSV file. Finally, it loads the list of already processed files from the CSV file, loops through each audio file in the control_train folder that has not been processed yet, transcribes the speech in the audio file using the transcribe_wav function saves the transcription to the CSV file using the save_progress function, and prints a message indicating that the transcription has been saved.

The transcribed data is saved in a .csv file in the following format:-

Final datasets prepared

• Dysarthria training audio - This is a collection of various speakers and their pre-processed audio, following the cleaning process outlined earlier. There are approximately 15,000 samples in this category.

• Dysarthria testing audio = This type of data differs from the above as it is not utilized in the training process. Instead, it is solely used to evaluate the accuracy of our trained model when introduced to unfamiliar data. With close to 5,000 samples falling into this category, it serves as a crucial component in evaluating the effectiveness of our model.

• The transcriptions.csv file - This and the dysarthria training audio are in sync i.e. no transcriptions were generated from the control audio for the corresponding dysarthria testing audio.

The Model

When I presented the challenge of developing an ASR for identifying dysarthric speech to GPT-4, it recommended utilizing the Wav2vec2 model through Python code. As I encountered errors while running the code on certain data, I provided them to GPT-4, which eventually gave back the code that effectively handled unusual data lengths, shapes, and modulo operations.

Here is the code that involves training the model:-

import os
import re
import numpy as np
import pandas as pd
import torch
import torchaudio
from torch.utils.data import DataLoader, Dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from torch.optim.lr_scheduler import ReduceLROnPlateau
from sklearn.model_selection import train_test_split
from torch.nn.utils.rnn import pad_sequence
from transformers import get_linear_schedule_with_warmup


def normalize_waveform(waveform):
    return waveform / torch.max(torch.abs(waveform))

def contains_nan(tensor):
    return torch.isnan(tensor).any().item()

def calculate_validation_loss(validation_loader, model, device):
    model.eval()
    total_val_loss = 0

    with torch.no_grad():
        for batch in validation_loader:
            input_values, labels = batch
            input_values, labels = input_values.to(device), labels.to(device)

            outputs = model(input_values, labels=labels)
            loss = outputs.loss
            total_val_loss += loss.item()

    model.train()

    return total_val_loss / len(validation_loader)

def custom_collate_fn(batch):
    input_values, labels = zip(*batch)

    # Remove the extra dimension from input values
    input_values = [iv.squeeze(0) for iv in input_values]
    labels = [l.squeeze(0) for l in labels]

    # Filter out empty tensors
    filtered_data = [(iv, l) for iv, l in zip(input_values, labels) if l.dim() != 0]

    input_values, labels = zip(*filtered_data)

    input_values = pad_sequence(input_values, batch_first=True)
    labels = pad_sequence(labels, batch_first=True)

    return input_values, labels



def get_last_part(filename):
    base_name = os.path.splitext(os.path.basename(filename))[0]
    if base_name[0] == 'C':
       base_name = base_name[1:]
    return base_name

def read_transcription_file(file_path):
    with open(file_path, 'r') as f:
        return f.read().strip()


class DysarthriaDataset(Dataset):
    def __init__(self, audio_paths, transcriptions, processor):
        self.audio_paths = audio_paths
        self.transcriptions = transcriptions
        self.processor = processor

    def __len__(self):
        return len(self.audio_paths)

    def __getitem__(self, idx):
        audio_path = self.audio_paths[idx]
        transcription = self.transcriptions[idx]

        # Load audio and preprocess
        waveform, _ = torchaudio.load(audio_path)
        waveform = normalize_waveform(waveform)
        input_values = self.processor(waveform, return_tensors='pt', sampling_rate=16000).input_values[0]

        # Process transcription
        with self.processor.as_target_processor():
            labels = self.processor(transcription, return_tensors='pt').input_ids[0]

        return input_values, labels



processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-large-960h")
processor.tokenizer.do_lower_case = True


model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-large-960h")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)


# Replace with your actual file paths and transcriptions
# load audio paths and transcriptions
audio_dir = "/kaggle/input/dysarthric-audio/dysarthric_train_pre-processed/dysarthric_train_pre-processed"

# Load the CSV file into a DataFrame
csv_file = "/kaggle/input/dysarthric-audio/transcriptions.csv"
df = pd.read_csv(csv_file, nrows=1500)

# Extract the audio paths and transcriptions from the DataFrame
audio_paths = df['file_name'].apply(lambda x: os.path.join(audio_dir, x + '.wav')).tolist()
transcriptions = df['transcription'].tolist()

# Filter out audio-transcription pairs with empty transcriptions or missing audio files
non_empty_pairs = [(audio, text) for audio, text in zip(audio_paths, transcriptions) if text.strip() != '' and os.path.exists(audio)]

# Split the filtered pairs into separate lists of audio files and transcriptions
filtered_audio_files, filtered_transcriptions = zip(*non_empty_pairs)

train_audio_files, val_audio_files, train_transcriptions, val_transcriptions = train_test_split(
    filtered_audio_files, filtered_transcriptions, test_size=0.1, random_state=42)

# Pass the filtered transcriptions to the DysarthriaDataset
train_dataset = DysarthriaDataset(train_audio_files, train_transcriptions, processor)
val_dataset = DysarthriaDataset(val_audio_files, val_transcriptions, processor)
train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True, collate_fn=custom_collate_fn)
val_loader = DataLoader(val_dataset, batch_size=4, shuffle=False, collate_fn=custom_collate_fn)


epochs = 5
warmup_proportion = 0.1
num_training_steps = len(train_loader) * epochs
num_warmup_steps = int(warmup_proportion * num_training_steps)
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-7)

scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps)

model.train()

for epoch in range(epochs):
    epoch_loss = 0

    for batch in train_loader:
        input_values, labels = batch
        input_values, labels = input_values.to(device), labels.to(device)

        optimizer.zero_grad()

        outputs = model(input_values, labels=labels)
        loss = outputs.loss
        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)

        optimizer.step()
        scheduler.step();
        epoch_loss += loss.item()

    val_loss = calculate_validation_loss(val_loader, model, device)
    print(f"Epoch {epoch + 1}/{epochs}, Train Loss: {epoch_loss / len(train_loader)}, Val Loss: {val_loss}")


model.save_pretrained("/kaggle/working/dysarthria_model_1")
processor.save_pretrained("/kaggle/working/dysarthria_model_1")

Explanation The code is a Python implementation of a Wav2Vec2 model used to build an ASR (Automatic Speech Recognition) tool for dysarthric people to transcribe the words spoken. The code is divided into several functions and classes that handle various tasks such as normalizing waveform, calculating validation loss, collating data, reading transcription files, and preparing datasets.

The processor and model is initialized with pre-trained weights from the Wav2Vec2 model. The DysarthriaDataset class is defined to load and preprocess the audio files and transcriptions.

The train_loader and val_loader are defined as DataLoader objects that load the train_dataset and val_dataset respectively. The train_loader and val_loader are used to train and validate the model respectively. The optimizer and scheduler is defined to optimize the model's parameters during training.

The model is trained for epochs number of epochs with each epoch having a num_training_steps number of training steps. The validation loss is calculated after each epoch. The model's parameters are saved after training.

That's a pretty nice explanation by Hashnode's writing assistant ✨.

The Training

It took some time to develop the code to its current state, working on it only during the last and current weekends. My focus was solely on instructing GPT-4 to solve errors in the code it generated, rather than brainstorming solutions. The shared dataset was not my initial starting point, as the original data lacked diversity and had low-quality transcriptions. To enhance the dataset's diversity, I prompted GPT-4 with additional information to split the data into training and testing sets while using a more robust library to improve transcription.

Initially, I used my laptop as my first training machine. I began with a sample size of 100, which later increased to 500. Unfortunately, I encountered strange results during training as the loss became "nan", and I felt I had hit a wall prematurely. However, the next day, I discovered that my code only read file names, not their content 🤦 . After fixing this issue, my losses produced numerical values.

I discussed this project with my friend with whom I collaborated back for the college project and he suggested to checkout Kaggle to get free powerful GPUs to train the model. I certainly needed to do that because at one time I tried training the model with 5k samples in my machine and after even 1.5 hours, the first epoch wasn't even logged 😅.

So I took my whole setup to Kaggle. This happened very early on in the process, so even on Kaggle, a lot of data that wasn't polished enough got uploaded only to end up with the polished one which I stated in the final datasets section above.

Kaggle allows you to use powerful GPUs with some user data confirmation. I used the GPU P100 on my friend's recommendation.

I began with a modest 100 samples, gradually increasing to 500, and found that training proceeded quickly. Thanks to Kaggle Playground, I could train 5,000 samples in just an hour.

To improve my results, I switched to the pre-trained facebook/wav2vec2-large-960h model. While I had initially used the facebook/wav2vec2-base-960h model, I found it struggled to handle larger sample sizes and often omitted poorly transcribed audio or empty strings in new audio samples.

The Results

As previously mentioned, my lack of domain expertise became a roadblock in my progress. This is not a new obstacle for me, as I faced similar challenges during my college days. However, I was curious to see how far GPT-4 could take me.

As I experimented with different no. of samples, I tested the trained model (which got saved in the training process) with different audio of dysarthric speakers. At one time, I thought that "Hey this might work" when I tested the following word:-

Command

• Facebook one transcribed its dysarthric audio as "C MANG"

• The trained one transcribed it as "COMA"

Initially, I believed this approach was effective. However, after testing the model with numerous input strings, I noticed a trend. It seemed that the model had learned to avoid having any gaps between transcribed letters. This could be due to my training the model solely on individual words rather than continuous speech.

Occasionally, the audio yields empty text or merely the first letters, despite the pre-trained model generating more letters appropriate but incorrect transcript.

I could only get the model to work with upto 2k samples. Anything more than that and the losses became negative/nan or the transcriptions worsened.

In certain scenarios, the trained model helped me to closely approximate the speech of dysarthric patients. For instance, if the patient spoke the word "INTO" with difficulty, the Facebook model produced "EN TOO" while the trained model generated "INTU". However, the output was not always consistent and I believe that modifying some parameters in GPT-4's instruction caused me to lose this capability after making some changes to my model.

There were instances of negative training and validation losses, indicating overfitting and a lack of learning in the model. This issue is exacerbated when more samples are being trained. To address this, I experimented with various factors such as learning rate, epochs, optimizers, and gradient normalization parameters. While reducing the epochs and learning rate resulted in non-negative values leaning toward 0, it also caused the model to transcribe inaccurately.

Upon questioning GPT-4 about my attempts, it recommended certain actions. However, I found it frustrating and unproductive as it failed to recall previous suggestions, resulting in a trial-and-error approach. Despite my repeated attempts to inform it of previous failures, I was met with the same unhelpful recommendations.

The Conclusion

Overall, I was attempting to make something work without the necessary domain knowledge. It became clear that GPT-4 can only be effective if the user has a good understanding of the subject matter they are utilizing it for.

This is where it shines:-

Unfortunately, I am facing a challenge where substantial research is required, and I lack knowledge not only about ASRs but also about the fundamental technology behind the Wav2vec2 model and the appropriate datasets required to solve my problem. Although I have a basic understanding of what I want, it is insufficient when dealing with a complex problem without an understanding of the limitations of the methods used to solve it.

Some doubts that crossed my mind:-

• Is Wav2vec2 the right model selection to solve this issue?

• How effective is a dataset comprising audio+transcription of single words in shaping and training a speech model, particularly one designed for continuous speech? Despite searching online for databases with continuous dysarthric speech, I couldn't find any good dataset. So, I downloaded a test file from the TORGO database, but it didn't work out. It only had single words or sometimes a sentence here and there, which didn't help me with what I was trying to do.

• Have the input values for the model been appropriately transformed? Is the padding method for variable-length audio inputs satisfactory, or does it require adjustments?

The Desire

I'm still interested to find a solution to this problem, but with a knowledgeable person in the ML space who can handle the domain-specific shenanigans. I wanted to evaluate a model like Wav2vec2 only to set up a good enough pipeline to try this with Whisper since it was trained on much larger data and I expect it to do better.

I used replicate to try a sample of Dysarthric audio with Whisper 2.0 for the "Command" word and it transcribed it as "come on" (temperature - 0.2)and "COMMENT!" (temperature 0.4). That's way better than what wav2vec2 came up with.

If anyone is interested, there is a blog post to fine-tune Whisper for multi-lingual audio:-

I thought maybe if I get consistent and plausible results with Wav2vec2, I could go ahead with Whisper and make something much better. I am not even sure if Whisper is made for something like this but we won't know unless we try. If you're someone who wants to solve this problem and actually can make logical/intuitive decisions regarding the whole problem statement, reach out to me. That's how I would like to solve it further.

Achieving actual results would have been super cool, but I refuse to waste my time relying solely on luck. When building something, understanding the reasoning behind most of the decisions is crucial. It makes it easier to maintain and build stuff upon it later.

Thank you for your time 🙏

But first the teaser of what's finally built:-2023 certainly feels like the year of AI. Post public reveal of ChatGPT, especially, the reach of this technological revolution has reached beyond the computer folks. My cousin used ChatGPT to help her with articles. David Guetta is trying out AI tools now:-2023 certainly feels like the year of AI. Post public reveal of ChatGPT, especially, the reach of this technological revolution has reached beyond the computer folks. My cousin used ChatGPT to help her with articles. David Guetta is trying out AI tools now:-2023 certainly feels like the year of AI. Post public reveal of ChatGPT,

2023 certainly feels like the year of AI. Post public reveal of ChatGPT, especially, the reach of this technological revolution has reached beyond the computer folks. My cousin used ChatGPT to help her with articles. David Guetta is trying out AI tools now:-

Microsoft just revealed the AI supercharged Bing to take on Google (who are playing catch up using Bard). For the first time, I have felt like a AI technological revolution is truly here at the consumer level.

Trying out something

With all the AI fluff spreading like wildfire, it was only natural to do a hands-on. Call it FOMO or call it curiosity, there are all sorts of projects out there right now built upon Open AI APIs either directly or using replicate or some other API gateway to make it a plug-and-play solution.

fsdfsdf

I thought I might probably try out something around images since it's where my creative interests lie the most. I have certainly played with midjourney, Dall-E and Stable-Diffusion to see the capabilities and midjourney, by far is my favorite.

Coming across articles and code around Q&A with GPT-3

So while exploring and following the AI developments, the quality of life improvement that I feel AI is going to bring is to turn the heat up on Q&A formats that have existed for a long time. Take ChatGPT for instance. It's such a good Q&A assistant. With it's pre trained data, it performs certainly well. Heck, I even used it to ship a line of code to production at Hashnode not so long ago.

The above article on dagster's blog is probably read by many folks till now. TL;DR version :-

This article explains how to build a GitHub support bot using GPT-3, LangChain, and Python. It covers topics such as leveraging the features of a modern orchestrator (Dagster) to improve developer productivity and production robustness, Slack integration, dealing with fake sources, constructing the prompt with LangChain, dealing with documents that are too big, dealing with limited prompt window size, dealing with large numbers of documents, and caching the embeddings with Dagster to save time and money.

GPT-3 is a powerful model or to be more accurate a set of AI models that serve various purposes. ChatGPT is based on GPT-3.5 and so better at stuff than GPT-3. It's soon going to be available as an API as well.

What are the ways to train a GPT-3 model ?

There are several ways to train a GPT-3 model, including fine-tuning, data augmentation, and vector-space search. Fine-tuning involves training the model on a specific dataset, while data augmentation involves providing additional data to the model to improve its accuracy. Vector-space search involves using a search engine to find the most relevant sources for a given query.

A lot of technical jargon again but as far as I understand, there are two ways I like to understand how you can extend a model’s capability to do something for you:-

• Fine-tuningaims to train a certain model with pre-trained data to get better at giving accurate answers to the user's provided prompts. Consider you want to fine-tuneGPT-3 model for certain documentation you have been working on. After going through the few articles and skimming through the fine-tuning docs, I got to understand that there are costs involved in fine-tuning a model using OpenAI. Also, models like text-davinci-003 cannot be used right now for it and so only base models can be used. Again, it's the lack of interest from my side here specifically that I haven't explored this fully. This probably might be the way to go to train on production data and get a model that performs exactly what you intend it to.

• Prompt engineering is the process of developing a great prompt to maximize the effectiveness of a large language model like GPT-3. It's like using a model but making it aware of a limited context and only asking it to answer based on that context. This awareness of context happens at runtime, unlike fine-tuning.

Here is another tweet and linked article I read recently which explores creating a Q&A model for documentation using the same prompt engineering:-

In fact, in its final Shoutouts section, the author has added references to the dagster article I shared above and more.

So I also wanted to explore the prompt engineering side of things. Langchain is one of the terms you would read or hear more about when touching this territory.

Langchain helps developers combine the power of large language models (LLMs) with other sources of computation or knowledge. It includes features like Data Augmented Generation, which allows developers to provide contextual data to augment the knowledge of the LLM, and prompt engineering, which helps developers develop a great prompt to maximize the effectiveness of a large language model like GPT-3.

In this article though, I do not dive into how langchain works. I am not even fully sure of the intricacies of how the math behind a lot of stuff behind all these AI stuff works but at the end of the day, it is all math. But one of the terms you will come across frequently when dealing with input data is embeddings. Embeddings help to represent words in a numerical form and can help us measure the similarities between these words. Think of it as a vocabulary that can fit a lot of words. So for big input data, you convert them into embeddings.

Here's is how I have understood prompt engineering to work:-

1. Prepare some data you want the AI model to be aware of.

2. Split that data into chunks of text and convert them into embeddings. (It's good to split data into chunks because OpenAI has limits on how many embeddings can be processed by its model as well but still, that limit is almost twice of if a raw string was given to the GPT-3 competition endpoint directly)

3. Calculate embeddings for the question and compare those with the input data embeddings. The closest ones are chosen to give us the context or the raw string that can be fed into the completion endpoint.

4. There is a prompt template involved so that the model can be made aware of what it needs to look for answers in and if it's not able to find it, it can say that it doesn't know the answer instead of giving the wrong one.

   Here is how a prompt template might look:-

   Answer the question based on the context below, and if the question can't be answered based on the context, say "I don't know"

   Context: {context}

   ---

   Question: {question}

   Answer:

A web crawler powered Q&A service with OpenAI

In the dagster article, there was a point in their Future work section that stated this :-

Crawl web pages instead of markdown. It would be relatively straightforward to crawl a website’s HTML pages instead of markdown files in a GitHub repo.

This resonated with me and while exploring all this AI stuff, specifically on the openai fine-tuning docs, at the bottom of the page, there is an examples section that contains links to the python notebooks by openai relevant to Q&A. So I went through the notebooks involved, starting with this:-

The above landed me here:-

But I think the most important finding was that openai has these cookbooks essentially. So I went to the root folder to see what all they have and that's where I landed on this:-

The above cookbook was added just a week ago and does exactly what I was wondering about.

In a nutshell, it did the following:-

1. Crawl https://openai.com/ and generate all the text files for each of the links crawled.

2. Generate embeddings from all the text files by chunking them first.

3. The whole question and data embeddings compare step.

4. Then trying to answer a question via a prompt template (exactly what I mentioned before as an example).

I installed python3 on my system. Then copied the relevant files from GitHub. After a couple of Stack Overflow searches and setting up the OpenAI API key (paid stuff), the program ran and did exactly what was shown in the python notebook online.

Then everything from there was me running the same program for a couple of different sites. Started with the Next.JS blog and docs, my blog and a couple of other sites.

All in all, I refactored the whole code to be more reusable and utility oriented in the following manner:-

• Allowing either to form the Q&A context on a single page or the crawled pages via a configurable recursive parameter.

• Modifying the folder and file-naming code to save data for each site that is crawled.

• Not crawling or creating embeddings again if already exists for a site and proceeding directly to the answer part.

• Creating a minimal flask service to consume the refactored code.

The whole refactor resulted in the following 5 python files:-

• web.py : Managed the whole crawling and saving text files from the crawled links.

• utils.py: All the utility functions being used across different files.

• embeddings.py: Managed the whole generation and saving of the embeddings.csv being created.

• answer.py: Managed prompt engineering bit of the code.

• server.py: The flask service wrapper over our controller function.

• index.py: The file containing the high-level code that runs when the flask service is hit.

    # index.py file
  
    from web import crawl
    from embeddings import generate_url_embeddings
    from answer import getAnswer
  
    def train(url, recursive=False):
        crawl(url, recursive=recursive)
        generate_url_embeddings(url, recursive=recursive)
  
    def ask(question, url, recursive=False):
        train(url, recursive=recursive)
        return getAnswer(question, url, recursive=recursive)
  

  Playing around with the created API

  • nextjs.org/blog : This was created using recursive:true so that we can crawl all the links and build the input data from there.

    • Question: In which version of Next.js, unoptimized:true got supported inside next.config.js ?

      Reply by API: Starting with Next.js 12, unoptimized:true got supported inside next.config.js.

    • Question: What are the latest updates to middleware?

      Reply by API: With 13.1, you can now return responses from Middleware, as well as set headers on the request. These API improvements give you powerful new flexibility to customize every part of the Next.js routing lifecycle. The experimental.allowMiddlewareResponseBody configuration option inside next.config.js is no longer required. You can now more easily set headers on the request, as well as respond directly without having to rewrite or redirect.

  • Fun fact: The TL;DR version of the dagster article I stated earlier was generated by this API and also the answer to the question on how we can train GPT-3 model and the bit about langchain.

  • https://vercel.com/legal/privacy-policy : Created using recursive:false to just get the text for this page.

    • Question: What is stated about data usage?

      We use data to administer and improve the Services, analyze trends, track users' use of the Platform, and gather broad demographic information for aggregate use. We also use Device and Usage Information to generate and process aggregated information, such as statistical or demographic data. We may also track the total number of visitors to our Platform, review Log Data, or track the number of visitors to each page of our Site. We may also use Location Information derived from IP addresses. We do not, however, collect or process End Users’ IP addresses through the use of our analytics Services.

  • https://vercel.com/legal/terms (recursive:false)

    • Question: Which terms are not in favor of the user?

      You agree that violations of this Agreement by yourself or any person or entity acting under your account will, in addition to any other remedies including criminal prosecution, result in the termination of your access to Vercel's website and removal (taking down) of all projects and deployments. In addition, violation of these terms or any of Vercel's policies may result in tracking information being stored to identify the offending person and permanent restriction from holding an account on Vercel's website and Services. We may shut down affected projects or deployments on the hobby plan in case of any delays or performance problems including, without limitation, those caused by a malicious attack on a project or deployment. We will notify you if your usage is unreasonable, and we may shut down and terminate projects and accounts that are creating an unreasonable burden on our infrastructure and/or business operations, as outlined in Sections 9 and 16.1 below. All fees are non-refundable, except as expressly stated otherwise in this Agreement.

    • Question: Which terms are in favor of the user? Summarize them in less than 50 words.

      The user is granted a personal, non-sublicensable, nonexclusive, nontransferable, limited license to use the Services for their internal business or personal purposes. They are also granted a license to use and reproduce their trademarks, tradenames and logos in Vercel's marketing materials and website(s). Vercel will abide by any written trademark usage guidelines provided by the user. Vercel will also provide thirty (30) days prior notice for any changes to the fees or Agreement.

  • Latest featured article on hashnode - https://iwtyo.today/cognitive-bias-as-a-software-developer (recursive:false)

    • Question: Create a TL;DR version of this article

      Recognizing our own cognitive biases is essential to our lives and work. Egoless coding is a great practice to cultivate, as it helps us to accept and move on from our less-than-perfect code. To identify our skill level, we can ask our colleagues for their honest opinion. We should also break down our core skills into their most basic components and assess our proficiency in each one. Lastly, don't be afraid to seek help if you think you may be suffering from Imposter Syndrome.

    • Question: What are all the ways to identify skills?

      Identifying your skills can be done by assessing your proficiency in each component of your core skill, asking your colleagues for their honest opinion of your capability, contrasting your point of view with others, recognizing the role others can play in helping you improve, building communication and relationships, acknowledging the skills that you have and learning the ones that you don't, and recognizing cognitive biases.

All right. I might have gone overboard by stating the examples but trust me, I have run this service on a lot of sites by now. Thus the bill:-

If you want to play around with the code, feel free to check out the following:-

Limitations

• As the partial author of this code, I am not aware of the internal workings of how the embeddings are logically compared. I would say it's a knowledge limitation that can be overcome by someone else who is good at python and general ML stuff or by me if I get into maths again. For me, most of the things are a black box and work on high-level concepts.

• The speed is a total bummer. It takes minutes to train a recursive:true site and 40-50s for a recursive:false one. Again the query to the completion endpoint with the saved embeddings can take between 10-60s . But if you're crawling a 20 mins article then it's not that big a deal.

• There is a cost associated with the creation of embeddings and query completion but probably much less than fine-tuning.

• Sometimes the training isn't good enough and results in a lot of "I don't know" responses even when it shouldn't have. Also, I have seen the model returning wrong answers in a few instances even with this prompt template.

Conclusion

This effort was purely done out of curiosity and touching the AI waters. I for one am excited to try out the Bing chat assist because it fundamentally can do all the above very fast and at a much greater precision. Nevertheless, this was fun to create. Python isn't my work language so my command of it isn't as good. But in the era when there is GithubCopilot, I guess the lack of command gets partially hidden by it.

The transformation of the tech industry by AI is what I am looking forward to.

Thank you for your time :)

There are days in my job when I am implementing a new feature, fixing a bug or documenting some stuff. Most days are occupied with this stuff.

But some days are all about digging into changelog, commits, issues etc to find the cause of some unexpected behaviour of the library/framework being used.

In this article, I will be sharing instances of when at work, I had to deal with the above.

Also, I am a Software Developer at Hashnode and my experiences are derived from working here. It's a crisp collection of debugging stories and tagged as part of

Although this article won't be eligible for the writeathon, it might help you steer yourself in the right direction while solving your next bug 😉.

1. Next.JS middleware not working suddenly as expected

I wrote a whole article on this one for our engineering blog last year. You can check it out below:-

TL;DR :-

• Went to Github to check if in the last 24 hours, Next.js shipped something related to middleware.

• Saw a breaking change in how query parameters behave within middleware in the latest canary release.

• Realised that middleware is in Beta and isn't bound by semantic versioning. That's why the unexpected behaviour.

• Fixed the code within middleware according to the latest changes.

2. Next 12 -> Next 13 upgrade causes product tour to fail

• We recently upgraded the codebase powering hashnode.com from Next 12 to Next 13.

• On production, we observed that the onboarding product tour for new users isn't working correctly anymore.

• Everything works correctly in local development. But I was able to replicate the same by running the built output locally.

• This meant something was wrong with the built output.

• I went to the library's Github Issues tab and started searching for stuff related to Next JS which led me to find the following issue :-

• The following comment by the maintainer helped in knowing the cause and fix for the issue

  

• Jumping to the discussion thread above, the swc maintainer had already acknowledged the issue which whose patch will land in version Next 13.0.7. The PR with fix:-

• Till then just adding swcMinify:false to next.config.js solved the issue and the built output was correct. Starting from Next 13, swcMinify was set to true by default unlike the previous versions and that's why we only saw this behaviour on the update.

3. The Blog Stats graph started indefinitely shrinking on browser zoom at 90%

• The first step for me here was to simply google the above title as it is which landed me here

• The above discussion led me to the following PR which was opened to fix it

• This fix landed on the version 4.1.1 of the chart.js library. So I simply updated our library version to the latest one (4.2.0) since that already has the patch.

4. withAxiom API wrapper increased our Vercel serverless execution GB hours

• We use axiom for logging purposes at Hashnode.

• In early January, we wrapped one of our /api handler with the withAxiom HOF.

• We observed that since then our serverless execution GB hours SKYROCKETED!!

  

• Further, we saw how the internal implementation of withAxiom was and found out that there were a couple of awaits that dealt with flushing the logs and increased our GB hours.

• So we removed that wrapper function and saw the GB hours stabilise again. We already informed the maintainers about it and they are thinking of a solution to rectify the behaviour.

5. unoptimized:true not working in next.config.js as we expected

• If you go to next.js docs, you will find how one can disable the image optimization by setting unoptimized:true in the next.config.js

• But the above wasn't working for us in one of the projects but surprisingly working in another.

• The difference between both was the version of Next JS that we were running.

• The one where we had Next 13, the above worked. The one where we had Next 12.2.3, it didn't.

• So I dug up the releases by going to Github and found out that the support for stable unoptimized landed in version 12.3.0. Before that, you can only replicate the same functionality by setting the flag inside the experimental object of next.config.js.

• Before 12.3.0 :-

    // next.config.js 
  
    {
    experimental:
        {
          images: 
           {
            unoptimized:true
           }
        }
    }
  

• After 12.3.0 :-

    // next.config.js 
  
    {
      images: 
         {
          unoptimized:true
         }
    }
  

• Validated the same by going to the changelog blog by Vercel.

• So just updated the non-working project to 12.3.0 and the unoptimized:true started to work as we intended.

Conclusion

It does feel like detective work. You are going through clues, clicking a link, and getting redirected to something similar. Hopping from one issue, commit to another, you finally find the line of code or PR which resolves it all for you.

Also, expecting oneself to do this on Day 1 of their career or a new job is probably not the smartest strategy. It takes time to get into the flow of finding patterns and I think past experiences just keep adding up to it.

Feel free to share similar instances of work in the comment section!

Thank you for your time :)

It took me some time to come up with this title.

If the above doesn't make sense, then we recently launched our new editor experience which we like to call Neptune. More about it below:-

And this is my first article using that editor (Not first technically since I am sure I have written a bunch of drafts and articles while testing it). But yes, first public article for the readers 🎉 of my blog.

But what is this article about?

Well, I completed one year at Hashnode a month ago and this article is all about reflecting on that experience. I already shared my experience in another article I wrote when I completed 6 months at Hashnode. So, to catch up, kindly read the below:-

Now let's see what else was on my plate over the last 6 months:-

Twitter Connect

Did you know that you can connect with your Twitter tribe on Hashnode as well? We make it easy for you to find those folks and follow them instantaneously.

This was fun to build. I got to touch both the front and backend layers to build this. We started with one design but then shifted to a much more UX-friendly one and worked on how the user recommendations should turn up. It was fun to collaborate with Megha Pathak , Girish Patil and Sandro Volpicella on this.

You can read more about it below:-

Migrating our markdown editor from the old codebase to the new one

I think this is the first time I got to lay my hands on the markdown editor. This is easily one of the most critical parts of Hashnode experience and migrating it was certainly fun. I got to collaborate with Girish Patil and Rajat Kapoor on this one. And unlike the hashnode.com migration experience, this was much better planned and executed. The important part of that execution was documenting our goals, findings, and blockers etc everything on regular basis and communicating effectively on expectations. Also, this was probably the first time, we had an internal bug bash which went smoothly and made us much more confident in shipping it to our users.

But why was this migration needed in the first place?

Well, we are moving away from our legacy codebase steadily to make use of the capabilities the new one has to offer. We wanted to make the user experience feel much faster when they landed on our editor. And we saw a massive improvement in page load speed since earlier we had to proxy our editor and the extra hop cost us seconds. We not only migrated but also fixed some bugs in the markdown editor experience along with enriching the UI for it. Overall this was a solid A+ team effort.

Improving blog search on Hashnode and personal blogs

We introduced a new tab called Blogs which makes it easier for our users to search for individual/team blogs powered by Hashnode. Along with this, we also improved the article search within each blog.

Adding the capability to import dev.to posts as drafts

This was requested by many of the users of our platform and we finally prioritized this. The groundwork for this was already laid down by our engineers since we supported this for RSS articles. But it was still fun to work with nevertheless because the UX was much better than before. Now the users didn't have to worry about their old articles getting posted after import. They had the choice to publish them as drafts or posts.

Migrating Draft Preview to Blogs

The ownership of each article residing on your domain certainly feels good and Hashnode makes that an easy experience for the user. But there was

We make sure to sync your changes to a draft when you write something. But the previews of that draft used to still reside on hashnode.com itself. So when a user had to preview their draft article or someone else's, they landed on hashnode.com/preview/[draftId] and not in the writer's domain. So we had to make this experience seamless like our published articles.

I showed my interest to pick this up because I had a brief idea about the technical complexity of migrating this from our old codebase to our blog codebase. And after being part of two big migrations, I had enough experience to execute this one independently. And while the implementation went well, I think this got sidetracked in code reviews because we had other priority items to ship. So this stayed in code review for a whole MONTH!!. But I think everything worked out well since this affected not just our web experience but also how the mobile App flow would work with these new changes and this is probably the first time, I got to work with Mohd Shad Mirza as well.

Even before this got shipped, I worked on another task to ensure that a user can only create an article on hashnode if they have a blog. We wanted to push for content ownership here and unlock some capabilities that can only happen if the draft article is previewed on the user's domain (Support for custom widgets which we shipped recently for draft previews).

Overall, understanding the problem statement, executing it and tackling the edge cases was enjoyable while implementing this.

Worked on something that we couldn't ship to users

There are times when you might have spent working on something and even witnessed its completion but it couldn't go live due to a change of scope/priorities. This was a total revamp of one of our existing features where I got to collaborate with Sai Krishna Prasad Kandula where he did most of the heavy lifting. It was fun since we whiteboarded the flow diagrams along with possible edge cases and the executions. It started as a growth experiment but soon we realized that the scope was much broader but we managed to get it all working. The work is not discarded yet but it's also not made public to our users. Let's see when it sees the light of day 🤞.

The new WYSIWYG Editor Experience

I feel this is the highlight of my work at Hashnode in the past year. I collaborated with Girish Patil , Rajat Kapoor and Pritam Paul on this. As good as the editor migration experience was, this was a notch above. We solved for unknowns here and that makes a difference in the impact. There is so much to write about this but that would be better as a standalone blog article.

Hashnode Retreat

This was such a great experience. Met most of our team members finally and had a fun time with them at Phuket. Again, this could use another article. But here is a tweet by our PM Sumudu Siriwardana showing our happy faces:-

Alright in the last one, I am just photobombing and confused.

Few Bundled Things...

Debugging

I like to debug issues. That's why I probably still try to solve stuff at Stack Overflow even though I am sure, ChatGPT will make me obsolete soon 🥲.

There are occasional bugs that as devs we have to solve and after solving the middleware bug which resulted in this article on our engineering blog, I faced another one where one of the drafts by a user crashed our article creation API (legacy) and also didn't let the user load that draft preview. It took me an entire day to figure out the problem here after even an hour of pair debugging with Girish Patil .

The solution to this problem was very straightforward but finding the cause was not. It was perplexing, to be honest.

But sometimes, digging deep into such perplexing behavior can give us better returns than our assumptions about the code.

Code reviews & Documentation

Besides debugging, code reviews form an important part of our dev routine and we are trying to solve for speeding up that process. As a startup, we have to move fast while making sure we don't break things. And I feel code reviews are that one point that decides a lot of it. To fasten up these reviews, I have penned down my experience in form of some guidelines we can follow as devs to ensure basic coverage for all relevant parameters before we raise a PR for review. So a code reviewer doesn't need to do the rework here.

Down the line, I hope we automate many parts of it and that the improved DX helps us move fast and confidently. Also, code reviews help me collaborate with more folks than I can if I just stick to writing code.

Interviews

Being able to recruit for Hashnode is another experience I enjoy. We are welcoming a new engineer soon from those sets of interviews. Assessing candidates who are passionate about what they do and what they bring to the table is a constant learning in these interviews. So, I continue to look for folks who can help us scale Hashnode to the next level.

Self Assessment

After completing 6 months at Hashnode, I scheduled calls with our co-founders to talk more about what my career aspirations look like and what their assessment of me is. I would like to thank my Dad here for pushing me to have these discussions. Due to those, I have a much cleaner vision of what moving forward one step ahead looks like for me.

Personal Commitments

Okay, last time I was boasting about my fitness routine but this time I gained 4.5 kgs back again after losing more than 10 kgs 6 months ago.

But before that, it's good I went on a trek with my friend and it turned out to be an impactful learning experience. We got featured here:-

Wrapping Up

It's been rare that I didn't have to collaborate with Sandeep and Syed Fazle Rahman on most of the things mentioned above. These two care deeply about what's being built and bring their own founder experience to solve problems with us. We sometimes find Sandeep building something as a side quest in a couple of hours/days and Fazle implementing a bunch of design changes that we didn't even know about. They are still builders at heart.

The experience of ownership, impact and collaboration has surely increased since I shared my last one and I see that as a win.

Thank you for your time :)

I have been meaning to try out xstate for long now by building a stopwatch example using it. For functionality, I referred to iOS's stopwatch in their native Clock App.

But why stopwatch ? Well, I was once asked in an interview to build the same and I struggled with it. For me, the struggling bit was visualizing the states the stopwatch can be. This is the code I submitted post interview (the interviewer was very considerate and told me that though I couldn't complete the implementation within time span, I can submit the solution later.)

function StopWatch() {
  this.timerStart = 0;
  this.timerEnd = 0;
  let time = { hours: 0, minutes: 0, seconds: 0 };
  let interval;

  this.setTime = function () {
    if (this.timerEnd) {
      time.seconds = (this.timerEnd / 1000) % 60;
      if (time.seconds === 0) {
        time.minutes += 1;
      }
      if (time.minutes === 60) {
        time.hours += 1;
        time.minutes = 0;
      }
    }
  };

  this.getTime = function () {
    return `${time.hours}:${time.minutes}:${time.seconds}`;
  };

  this.start = function () {
    this.timerEnd = this.timerStart;
    console.log(this.getTime());
    interval = setInterval(() => {
      this.timerEnd = this.timerEnd + 1000;
      this.setTime();
      console.log(this.getTime());
    }, 1000);
  };
  this.stop = function () {
    clearInterval(interval);
    this.timerStart = this.timerEnd;
  };
  this.reset = function () {
    clearInterval(interval);
    this.timerStart = 0;
    this.timerEnd = 0;
    time = { hours: 0, minutes: 0, seconds: 0 };
  };
}

Not my best code and actually doesn't feel intuitive enough. There are still no states for the stopwatch but merely functions that can start, stop or reset the timer and also redundant variables. So we will tackle the non-intuitiveness of this solution using xstate.

Reference Point

Here is what the iOS stopwatch looks like in it's initial state

Here is what the iOS stopwatch looks like in it's running state

Here is what the iOS stopwatch looks like in it's paused state

Using the app helped me land at the possible states faster.

Code Playground

Now before we proceed, I had no clue how the whole API works to create a state machine using xstate. So I went to their docs and landed on the stately visualizer where one can load the default example (a machine to visualize possible states when you do a fetch operation). One can just click on the Visualize button and also see a neat Statechart diagram on the screen depicting the actions and states of the machine.

So I begin with modifying this existing machine to create the stopWatchMachine.

Note :- The default code playground was in TypeScript and I stuck to it so you can ignore type specific tokens in the code I am going to share further.

Building the stopwatch without lap feature

In the screenshots of iOS stopwatch feature, you will notice that there also is a Lap button which is used create laps to measure in-between time intervals. We will get eventually to that but for starters, let's focus on start, stop and reset functionality.

Alright then, let's start with laying out some code now :-

Initial State

import { createMachine } from "xstate";

interface Context {
  elapsedTime: number;
}


const stopWatchMachine = createMachine<Context>({
  id: "stopWatch",
  initial: "initial", // Note this can be anything - I just like to call it initial itself.
  context: {
    elapsedTime: 0
  },
  states: {
    initial: {
      on: {
        PRESS_START: "running"
      }
    }
}
})

Explanation :

• initial denotes what our starting state is going to be when this state machine is created.
• context is an object which holds the variables on which we actually want to operate. So in our case, we care about how much time has elapsed since you hit start on your stopwatch. We are measuring the value using the elapsedTime variable.
• Now comes the states object which really forms the basis of the whole statechart paradigm. Our first state in it is initial which itself is an object containing an on key. This on key contains key-value pairs of event : state to transition when that event happens i.e. when the PRESS_START event happens, we want our state machine to transition to running state.

Note - Don't worry if you don't see anything like running state now. We will get to it in the next step. If you try to click on Visualize button at this point, the code playground will complain about running state not being present.

Running State

import { createMachine, assign } from "xstate";

interface Context {
  elapsedTime: number;
}


const stopWatchMachine = createMachine<Context>({
  id: "stopWatch",
  initial: "initial",
  context: {
    elapsedTime: 0
  },
  states: {
    initial: {
      on: {
        PRESS_START: "running"
      }
    },
    running: {
      on: {
        PRESS_STOP: "paused",
        TICK: {
          actions: assign({
            elapsedTime: (context) => {
              return (context.elapsedTime += 20);
            }
          })
        },
      },
      invoke: {
        src: () => (cb) => {
          const interval = setInterval(() => {
            cb("TICK");
          }, 20);

          return () => {
            clearInterval(interval);
          };
        }
      }
    },
}
})

Explanation : Sorry for bombarding you with a lot of code in one step but that's really what's happening while our stopwatch is in running state.

• Inside the running state, we have a on binding for PRESS_STOP which should bring the machine in the paused state.
• Now comes the part where we start thinking about how to change the elapsedTime variable in a timer fashion. We know that a clock/timer works on basis of a certain TICK which happens every few milliseconds. So as soon as we enter this running state, we want to start a timer. To do this, we need to setup the invoke property first.
• One of the ways to achieve interval ticks in JS is via the setInterval API. Instead of modifying the elapsedTime within invoke object, we will emit a TICK event every 20 milliseconds (chose this for precision) using the setInterval API. The cb is supplied by xstate and is used to send any events to the parent (in this case - running state is the parent). Also that clearInterval bit is not necessary but it's good to perform memory cleanups.
• Alright, so now we need to listen for that emitted TICK event inside the running block also. Inside this TICK event, we want a certain action to be performed. This action should assign our elapsedTime context variable a certain value. So we simply increment context.elapsedTime by 20 on each TICK and return it.

Paused State

import { createMachine, assign } from "xstate";

interface Context {
  elapsedTime: number;
}

const stopWatchMachine = createMachine<Context>({
  id: "stopWatch",
  initial: "initial",
  context: {
    elapsedTime: 0
  },
  states: {
    initial: {
      on: {
        PRESS_START: "running"
      }
    },
    running: {
      on: {
        PRESS_STOP: "paused",
        TICK: {
          actions: assign({
            elapsedTime: (context) => {
              return (context.elapsedTime += 20);
            }
          })
        },
      },
      invoke: {
        src: () => (cb) => {
          const interval = setInterval(() => {
            cb("TICK");
          }, 20);

          return () => {
            clearInterval(interval);
          };
        }
      }
    },
   paused: {
       on: {
        PRESS_START: "running",
        PRESS_RESET: {
          target: "initial",
          actions: assign({
            elapsedTime: (context) => {
              return (context.elapsedTime = 0);
            },
          })
        }
     }
}
}
})

Explanation :

• In the paused state, currently, a user can press start to resume and go back to running state or a user can press reset to go back to the initial state. The on bindings depict the above statement in a declarative fashion.
• A few more things happen inside the PRESS_RESET event binding. Here we want changes to elapsedTime as well as a state transition to initial when this event gets emitted. context.elapsedTime is reset to 0 using an action.

Result

This partially completes the building of our state machine which can help us perform start, stop and reset operations of our stopwatch. We can create a stopWatchService like so and start using it :-

import { interpret} from "xstate";

const stopWatchService = interpret(stopWatchMachine);
stopWatchService.start();

// Now you can emit the events
stopWatchService.send({ type: "PRESS_START" });

(async () => {
  // delay is a promise wrapper on setTimeout
  await delay(2000);
  stopWatchService.send({ type: "PRESS_STOP" });
  await delay(2000);
  stopWatchService.send({ type: "PRESS_START" });
  await delay(2000);
  stopWatchService.send({ type: "PRESS_STOP" });
  await delay(2000);
  stopWatchService.send({ type: "PRESS_RESET" });
  await delay(2000);
  stopWatchService.send({ type: "PRESS_START" });
  await delay(5000);
})();

In case you want to listen to the state changes (which is needed if you want to consume the elapsedTime value or see what the current state of machine is), you can do so using callback inside onTransition function :-

const stopWatchService = interpret(stopWatchMachine);
stopWatchService.onTransition((state)=>{
console.log(state.context.elapsedTime, state.value)
})
stopWatchService.start();

// Start emitting events as shown in above code block

Building the stopwatch with lap feature

Before be proceed, here is how the Lap UI looks in the iOS app :-

You can see that once a user hits the Lap button, an entry of that lap is added to a list as shown below the timer. So this means that besides elapsedTime, we also have to maintain a laps context variable which will start as an empty []. Also Lap button is only enabled for the user when the stopwatch is in its running state. This gives us enough info to add the lap feature to our state machine like so :-

import { createMachine, assign } from "xstate";

interface Context {
  elapsedTime: number;
  laps: Array<{ startTime: number; elapsedTime: number }>;
}

const stopWatchMachine = createMachine<Context>({
  id: "stopWatch",
  initial: "initial",
  context: {
    elapsedTime: 0,
    laps: []
  },
  states: {
    initial: {
      on: {
        PRESS_START: "running"
      }
    },
    running: {
      on: {
        PRESS_STOP: "paused",
        TICK: {
          actions: assign({
            elapsedTime: (context) => {
              return (context.elapsedTime += 20);
            },
            laps: (context) => {
              const laps = context.laps;
              const latestLap = laps[laps.length - 1] ?? {
                startTime: 0,
                elapsedTime: 0
              };
              if (laps.length === 0) {
                laps.push(latestLap);
              }
              latestLap.elapsedTime = context.elapsedTime - latestLap.startTime;
              return laps;
            }
          })
        },
        PRESS_LAP: {
          actions: assign({
            laps: (context) => {
              const laps = context.laps;
              const newLap = { startTime: context.elapsedTime, elapsedTime: 0 };
              laps.push(newLap);
              return laps;
            }
          })
        }
      },
      invoke: {
        src: () => (cb) => {
          const interval = setInterval(() => {
            cb("TICK");
          }, 20);

          return () => {
            clearInterval(interval);
          };
        }
      }
    },
    paused: {
      on: {
        PRESS_START: "running",
        PRESS_RESET: {
          target: "initial",
          actions: assign({
            elapsedTime: (context) => {
              return (context.elapsedTime = 0);
            },
            laps: (context) => {
              return (context.laps = []);
            }
          })
        }
      }
    }
  }
});

Explanation : Holy moly, the state machine is quite bigger than before. Yes because we want to deal with setting the laps context variable on appropriate machine states. First let's see the introduction of PRESS_LAP event which can happen when machine is in running state. If that event happens, we don't transition to any other state. We just push a new lap to the the laps list inside our action block. Our lap data structure is an object which has two keys -

• startTime - This will be equal to the total elapsedTime on the timer when the PRESS_LAP event got emitted.
• elapsedTime - This will be the elapsedTime for that lap. We will see its derivation next.

Now, we already have a TICK event that happens every 20 seconds in the machine's running state. So now we also handle the laps context variable in the TICK event's action block. We only want the latestLap to keep respecting the dynamic timer. So we check the last element of laps list. If it's not there (for first lap), we insert one ourself with startTime as 0 and elapsedTime as 0. For each latestLap, we need to determine its elapsedTime. This can be done by subtracting the latestLap's startTime (Remember we set it when the PRESS_LAP event got triggered) from the total elapsedTime of the timer. After setting the latestLap, we return the laps object. Also in our paused state, we have the PRESS_RESET event definition. Earlier we only reset the elapsedTime to 0. Now we will also reset laps to [].

Final Result

This finally completes the building of our state machine which can help us perform start, stop, lap and reset operations of our stopwatch. Interpreting and starting the machine will remain same.

Following is the visualizer diagram of our final state machine :-

Also here is a codesandbox implementation with UI as well. The UI is not polished and doesn't do justice to iOS styling. It's just to see our stopWatchMachine powering an UI :-

Closing thoughts

It's the declarative API of xstate which really abstracts that nested if/switch case complexity when one is building an application. Once the machine was setup, I liked how deterministic the behaviour was. Overall it was fun to scratch the surface of xstate by building this example. There also is a low-code way of making the statechart first which automatically create a machine for you. Haven't tried that but that sounds very cool.

Thank you for your time :)

A Bit About My Work

Most of my work has been around frontend before joining Hashnode. I started with Wipro and then joined Altran (acquired by Capgemini). Picked up few learnings from being active on Linkedin (good old linkedin), contributing to Stack Overflow and making some side projects.

Coming from a service based MNC background of 3.5 years, Hashnode, being a startup, is a breath of fresh air in many aspects.

I have key highlights to share around my experience and I will keep them brief :

Being Part Of Hashnode Community Revamp

My first task or to be precise, a series of tasks to do as I joined Hashnode was to become part of the hashnode.com revamp. Trust me, there were moments where I felt like we're almost done but it just got further pushed. But I liked that we didn't compromise on the quality aspect of it. From frontend to backend, we did iterative improvements, discarded designs and re-approached with better ones, worked on performance a lot in the last days. This whole thing was literally my first 3.5 months at Hashnode but I am glad I got a chance to be a part of it. Got to learn a lot from the amazing folks here. Besides the technical aspects of learning, I understood some of design and SEO shenanigans and now try to incorporate these learnings in my side projects (Prings is the latest).

Creating Fun Side Project - JinglyPuff

So amidst the community platform revamp, Syed Fazle Rahman (CEO and co-founder of Hashnode) had an idea to make a very small fun project around Christmas theme. JinglyPuff was born out of it and was really my first visible contribution as part of Hashnode since the revamp was still a secret 🤫.

First Fully Owned Feature - Profile Hover Cards

Though community revamp had a mix of many tasks, most of those involved replicating existing functionality in a better way. Unlike those, this was my first feature development with full ownership of frontend and backend at Hashnode. Was very excited when this got shipped and yes, within 5 seconds I noticed a UI bug in prod but patched it before anybody noticed it (hopefully) and gave myself a pat.

Participating In Code Reviews

Before Hashnode, I have always had my code reviewed by seniors. Here we don't have that kind of hierarchy. I like to participate in them and make sure we ship quality stuff. We are still figuring out how to solve relevant stuff with tooling and have less cognitive load for developers. Besides the technical aspect of it, I like to participate in code reviews since they are one extra point of communication in a remote setup. LGTM.

Hiring for Hashnode

As you know we are expanding our team and that means having many applicants and interviews. I have never been at the interviewer side of the process and here I got this opportunity to interview folks. Our process is straight forward and consists of take home assignments. Reviewing those assignments and forming follow up questions and discussing tradeoffs with the candidates have been a learning in itself. Interesting folks have recently joined us at Hashnode 🥳 and are already creating waves.

Pitching Myself As Part Time Video Editor

From my first day, I made sure that folks at Hashnode know that I like to do creative stuff by letting Tim Cook wish them Good Morning as I unbox the work macbook

But yes, since then I go around jokingly pitching myself as part time video editor for Hashnode. I mean you gotta see the jinglypuff launch video :

Learning New Jargons And Acronyms

OKRs, CTRs, MAUs, investor jargon and what not are some stuff that you can easily pick up when working at a startup. Just today I learned what ymmv (your mileage may vary) stands for. Trust me I am not a very good acronym/jargon person. I totally google them most of the times but when you are in a fast growth startup, it's normal to come across these. I am pretty sure I know the least.

Making Memes (very important)

I think our education system should now include a whole chapter on memes. I maintain a folder called memes in my work laptop and I have created 5 till now. Hoping to kid around more and inspire people. Here is the latest one :

I mean, too much of even code reviews ain't good.

Brainstorming With Team Around Documentation, Guidelines & Engineering Practices.

This has increased many folds now and I love it. Every team member brings something to the table and so with more folks joining in, we have had some good discussions around engineering guidelines, documentations and processes. Also we have implemented a subset of these and hopefully by the end of the year, we would have much stronger foundations here.

My First Article Landed On Hashnode's Official Engineering Blog

Just this week, I got to publish the following article on our engineering blog. It feels good to be able to talk about your engineering efforts and document them and simultaneously represent your organization. Funnily enough, this is my first featured article on Hashnode 😎.

Pizza Monday

We had our first Pizza Monday (on a Tuesday) 2 weeks ago where I concentrated a lot on my pizza and had fun playing skribble with the folks.

Engineering is One Aspect Of It

At Hashnode weekly sync ups, all teams gather and share what they achieved this week, what's in work for the next time and more. Besides our web and mobile team updates, we discuss company's growth, investor updates, incidents & recovery insights, marketing team updates, management updates, design team updates, content team updates etc. Due to this intersection of information, there is also an opportunity to learn something more. I personally learned that even though Hashnode is powered by a lot of tech, the need to scale is very much dependent on how much our users love it. And engineering alone can't solve that. That's why we have these amazing folks at Hashnode who come up with strategies and design that drives engagement and ultimately makes sure the community thrives.

Making Progress

When I came to Hashnode, I identified myself as a frontend developer but now I am steadily moving towards my own vision of being a Software Engineer. And much of the credit for this goes to all the folks at Hashnode. There always is a helping hand around and that matters a lot to me from mentorship perspective. Nobody here sets a limit on what you can or cannot do. Not that I am going around learning every possible thing (personally can get counter-productive) but with an iterative approach, I feel I am progressing and that's important. In fact, as part of that progress, I undertook migration of one of the core components that powers search on Hashnode and a detailed article would soon land on our official engineering blog. So stay tuned ✨. Here's to unlocking more potential together

Personal Commitments

I mean it's a startup but it's flexible. You get to decide your work timings. You can take out time for stuff that's important to you and for me, even before I joined Hashnode, I was following a set routine to exercise and that continued after I joined it. I was able to attain a good health goal this year along with my work commitments and I see that as a win.

Wrapping Up

So these were the highlights of my 6 months journey with Hashnode. I still remember creating an account at Hashnode in 2020 but not really using it until 2021 and within one month of usage, applying for the job and now it's a part of my daily routine, sometimes as a developer and sometimes as a blogger.

Here are some special mentions as part of this experience - Vamsi, Megha, Girish, Sandeep, Fazle, Sandro

Looking forward to add more people to the list and collaborate 🤝 with them.

Also, we are constantly iterating on what Hashnode means to people and organizations and if that excites you, apply at our careers page.

See you soon. Ciao 👋

but then...... you get a $5.90 bill payment reminder for the nth time due to a s3 bucket you made on DigitalOcean Spaces that merely has 500MB data.

That's it. You have had enough. It doesn't make sense to pay at all for such a small storage. You got to put all your assets somewhere else. But before that let's have a little flashback :-

So more than a year ago, I participated in a hackathon organised by dev.to in collaboration with DigitalOcean. We had to make an app that has to be deployed on DigitalOcean's App Platform. So I made animeccha.com which is documented as a series of blog posts starting here. The images being served are powered by imagekit.io, the deployment by DO's App Platform and storage of those images in s3 bucket in DO's Space (this was free for starting 2-3 months due to $100 credit you get as free trial).

Now back to the present :-

My Image transformation was already powered by Imagekit but the storage was in a s3 bucket far far away (well not really, they have edge CDNs). So, I deleted the s3 bucket and just used the media library given by Imagekit as part of free plan with 20GB of monthly BANDWITH!!!. Lol should have done that in the first place.

So just by switching to it, I started saving $5.90 a month. Srsly, why was I even paying. Such laziness, much wow. But I did one more thing that was not required at all. I mistakenly also deleted my App Platform deployment so animeccha.com was now a template site of L'ORÉAL Products. No srsly, that's what my domain vendor thought of showing as a fallback.

Anyhow, I decided that my deployment platform should also shift and I chose Vercel because I have already used their services and find the setup relatively easy. As this project also has it's custom domain - animeccha.com, I had to switch my name servers from DigitalOcean's to Vercel's in my domain's DNS records. Cool the site was now up and running on Vercel, everything is back to normal.

But but....the developer urge to go further.

My App deals with images a lot and being a CSR (client side rendering) web app, the image loading even with placeholders wasn't the best UX. This is something I had in mind for long and now that I gained some experience in Next.js from my job (btw I work @Hashnode), I thought of migrating the whole thing to it and take advantage of their sweet Image component.

Their official docs have a great guide to do so and I followed it step by side. TL;DR - Getting rid of react-scripts, react-router-dom and installing next and updating package.json scripts. Also taking advantage of their pages route structure for flexible routing.

My old routes:-

• /home - Landing page

• /anime/:anime - Specific Anime Page

• /anime/:anime/:montage - Specific Montage of that Anime.

My new routes:-

• / - Landing page

• /:anime - Specific Anime Page

• /:anime/:montage - Specific Montage Page of that Anime.

The new Next.js powered routes made much more sense. / and /:anime got transformed to purely SSR pages powered by getServerSideProps and /:anime/:montage loaded the whole Montage component dynamically on client side since it required a lot of interaction with browser APIs (probably more time there and something else could be worked out but it's f9 for now).

And while this migration was happening, I realised something. I found modifying my CSS Modules powered styling less flexible now that I have used Tailwind at work. So yes I installed tailwind following this guide and changed most of my CSS to it except the Montage component since it had more moving parts and I wanted to keep them intact.

With all this, I deployed my changes in a new branch and raised a PR to the main one.

With two-three failing deployments, due to typescript errors and the fact that I forget to tell Vercel that what I am deploying is not a CRA project but a Next.js one, the project was finally live with much better UX.

But but....there could be

I also realised that both my / and /:anime pages are static and don't rely on server data. It was time to let them be treated likewise as well. Enter getStaticProps and getStaticPaths as part of SSG feature of Next.js. So I updated both the pages accordingly replacing getServerSideProps with the above functions and though not as such a visual difference, I somehow felt at ease now.

This was also my first time making efforts to migrate an existing side project. Though it's a very simple one and really doesn't have any customers and what not, refactoring it to make give a superior UX (even just to me) was a nice experience.

Thank you for your time :D

This year 2021, I wasn't sure I would be doing it again. But then I came across

and started digging into what all it's about.

Amazingly, I approached the things in a more structured fashion than the last year and I would like to take you through the first 4 contributions that qualified me for Hacktoberfest 2021 .

1. Numero Uno

Operational Transformation ?....sounds like a very sci-fi concept...doesn't it ?

A quick google search and I landed on the classic Wikipedia page describing what it was.

Operational transformation (OT) is a technology for supporting a range of collaboration functionalities in advanced collaborative software systems. OT was originally invented for consistency maintenance and concurrency control in collaborative editing of plain text documents. Its capabilities have been extended and its applications expanded to include group undo, locking, conflict resolution, operation notification and compression, group-awareness, HTML/XML and tree-structured document editing, collaborative office productivity tools, application-sharing, and collaborative computer-aided media design tools.[1] In 2009 OT was adopted as a core technique behind the collaboration features in Apache Wave and Google Docs.

Well, the above paragraph and a bit about it in the same article gave me a high level picture of the scope of this repository. It's collaborative editing and not limited to one Database or Editor implementation. Rather, this aims to allow multiple combinations of the either of Database/Editor as a part of otjs package. So you can have a firebase database with monaco editor or a firebase database with ace editor and this is just the beginning of possibilities.

As someone who hasn't really tinkered with either of used Database ,Editor or Operational Transformation tech jargons used here, this was very new and kind of exciting.

I randomly opened a couple of code files to see what's happening to only be clueless.

So I took another approach here. This approach has been talked a lot by folks all over dev communities when looking forward to contribute to a new OSS. It's reading the docs and figuring out the stuff. So naturally the README was the entry point for this which got me familiar with opening issues here (via discussions and not directly for this repo) and lead me to the CONTRIBUTING markdown file with the guidelines to make a contribution.

This is where I found my first way to contribute. Improving the existing docs, CONTRIBUTING markdown to be specific with some linguistic nuances and correct links. Does this make me understand the code better in any way ?

So why bother ?

It's the ice-breaker to start working on this repository. I ended up opening the following for this :-

The maintainer opened up a new issue

for me to fix and as per the CONTRIBUTING markdown guidelines, I made my first PR to this repository which got merged by the maintainer.

The PR 👇

2. Double Trouble

Well we have officially entered Hacktoberfest 2021 even though I didn't register until I got my 3 PR's merged I believe.

So I made a very small contribution to start with and the next one should at least give me more insight into running some examples using the packages in this repository.

And there was already a provision for examples here

But there was no README for the same which tells the user how to run this example. And this ultimately turned into another opportunity for a contribution.

This is the second discussion which took place

The maintainer opened up a new issue

and told me that I can pick it up if I want to. I gave it a try in my local setup and got the example running. After this, I created a README with instructions and few images that can give a new user enough information to run and validate the working of these examples. There goes my second PR to this repository.

The PR 👇

3. Third time's the charm

Till now, I actually didn't pick up any of the existing issues which were already up for grabs since most of them required technical know how. So it was time to try a small one to get my hands dirty with code.

I found the following issue fulfilling the criteria

what is happening

I basically had to add support for remote selection of text to be visible in Ace editor. Let's see what this means by example. Suppose two users are contributing to a Doc powered my Ace editor, now, when one of them highlights some text or places a cursor somewhere in the same DOC, the other should know where the cursor is and what's the highlighted/selected text.

And first I thought to myself that how is this task labelled small ? Wouldn't that need more lines of code ?

My question got answered when I compared the existing code for Monaco with Ace and saw that both have almost the same implementation with code for remote cursor/selection present as well. The only thing missing from Ace was a bit of CSS implementation and that was all that needed to be figured out. The good thing is that an existing implementation helped me compare and arrive at the solution quickly.

I submitted my PR and the maintainer appreciated the catch and fix.

The PR 👇

He also asked me if I can give a try to add support for a remote tooltip for Ace Editor. I did a quick comparison between Monaco and Ace implementation again and found that this would require implementing a new Class itself and should be addressed in a separate issue. The maintainer agreed and opened a new issue for the same.

4. May the Fourth be with you

The last issue helped me get familiar with a bit of code and resulted in creation of the above issue (labelled medium) which needed me to go deeper into Ace editor APIs. But I will be honest here, the editor doesn't have good docs (as far as I explored).

It was now required to implement a tooltip that lets the current user know the username of the remote user when typing. On paper, this sounds not too different from what might have gone into implementing a remote cursor/selection.

But when I looked at the existing code for tooltip implementation in Monaco editor, I saw that it has a concept of ContentWidget which allows for such UI widgets to be built and maintained easily and the developer used the same to create a new CursorWidget class which takes care of adding, updating and removing the tooltip. Here comes the interesting part that Ace itself doesn't have the concept of widget in the same sense.

There are two types of markers though which can be added by using addMarker or addDynamicMarker API. Initially I thought maybe the first one would be enough to help us achieve the tooltip functionality but since it's meant for static markers, which are added and removed each time using addMarker/removeMarker combo, it didn't fit the tooltip use-case. The second one i.e addDynamicMarker's didn't have good docs and I felt less confident in using it until I found the following tooltip implementation in another OSS

After reading the code and trying to implement the same kind of functionality a couple of times while ensuring to follow the CursorWidget template, I finally got around getting the tooltip implementation to work with Ace and proposed adding debouncing as well to the tooltip announcements for a better UX. Figuring out all of this took at least 2 full days and I was happy with the end result.

The PR 👇

If you notice the order of issues, you would see that this approach was progressive going from no-code to code contributions within 4 PRs. I know the maintainer from Linkedin and have interacted with him in DMs and comments a couple of times. So when I came across his post that he is looking for contributors to his new repo, I thought let's give it a shot !

While contributing this time, besides the problem solving and technical learning, I got to know about Semantic Commit Messages, --fixup flag, a gist of Github Discussions and creating issues from the same.

Besides the above 4 PRs, I contributed 3 more to the same repo with minor test updates, code refactoring etc. Also I came across the following to contribute JS interview questions and submitted 3 PRs for the same.

Overall I enjoyed the structured approach this time.

Thank you for your time :D

• level order traversal in binary tree
• zigzag order traversal in binary tree

And I like to solve them in a specific way which comes natural to me.

Binary Tree Level Order Traversal

Here is the code which works :-

var levelOrder = function(root) {
    if(!root)
        return [];
    let currentQueue = [];
    let nextNodes = [];
    const output = [[root.val]];
    currentQueue.push(root);
    while(currentQueue.length){
        const node = currentQueue.shift();
        nextNodes = nextNodes.concat(findNextNodes(node));
        if(!currentQueue.length && nextNodes.length){
            currentQueue = nextNodes;
            nextNodes = [];
            output.push(currentQueue.map(node=>node.val));
        }
    }
    return output;
};

const findNextNodes=(node)=>{
    const nextNodes = [];
    if(node.left){
        nextNodes.push(node.left);
    }
    if(node.right){
        nextNodes.push(node.right);
    }
    return nextNodes;
}

Explanation :-

• First we check whether root of binary tree exists or not. If not, simply return empty array [].
• Initialize a currentQueue and nextNodes array.
• Initialize a output 2D array where for each level we maintain an array consisting of the level nodes. Initially, the [root.val] will be the first valid entry. So output will be [[3]].
• And then push the current root node inside currentQueue. So currentQueue will be [Node(3)].
• Now, we will loop over the currentQueue until it is empty.
• Being a queue, we dequeue the first entry in it using shift operator on currentQueue.
• Now, we pass this entry or node(Node(3)) inside findNextNodes function whose purpose it to return us a list of children of the passed node.
• findNextNodes initializes an empty nextNodes array and pushes the left and right children of node if they exist and returns the nextNodes array. nextNodes will be [Node(9),Node(20)].
• Then the returned array is concatenated with nextNodes. nextNodes will be [Node(9),Node(20)].
• Now before the end of the loop, if currentQueue is empty and nextNodes is not, currentQueue starts referring the nextNodes array and nextNodes is reinitialized to []. Also, all the node.val values in the currentQueue are pushed in the form of an array to output. By this step, output will be [[3],[9,20]].
• This way the for each level we will successfully obtain the left to right level order traversal in the output array.

Binary Tree Zigzag Level Order Traversal

Before we see the code and explanation, it's important to note that we will be converting the above solution to obtain the current one. I would like to stress on this derivation because this is how I particularly enjoyed the process of arriving at this solution. We will do it in two iterations so that the process of arriving at this solution seems more intuitive.

Alright, here is the first iteration :-

var zigzagLevelOrder = function(root) {
    if(!root)
        return [];
    let currentQueue = [];
    let nextNodes = [];
    let isNextLevelEven = true;
    const output = [[root.val]];
    currentQueue.push(root);
    while(currentQueue.length){
        const node = currentQueue.shift();
        nextNodes = nextNodes.concat(findNextNodes(node,isNextLevelEven));
        if(!currentQueue.length && nextNodes.length){
            currentQueue = nextNodes;
            nextNodes = [];
            isNextLevelEven = !isNextLevelEven;
            output.push(currentQueue.map(node=>node.val));
        }
    }
    return output;
};

const findNextNodes=(node,isNextLevelEven)=>{
    const nextNodes = [];
    if(node.left){
        nextNodes.push(node.left);
    }
    if(node.right){
        nextNodes.push(node.right);
    }
    return isNextLevelEven?nextNodes.reverse():nextNodes;
}

Explanation :-

• As soon as you see the above code, you will realize that all it does is add subtle nuances to level order traversal's code.
• The first nuance is the isNextLevelEven variable. So let's discuss it. When we think about a zigzag traversal, it starts on level 1 i.e. root node from left to right and then on level 2 from right to left and the again from left to right on level 3. So there is a alternating state that's being introduced. Alternating state can be depicted using a boolean and that's what we are doing here. If root is level 1 i.e. an odd level, then the next level would be even. This is why I have initialized isNextLevelEven to true in the starting of this code.
• Before the start of the loop, output will be [[3]] and currentQueue will be [Node(3)].
• Now, we will loop over the currentQueue until it is empty.
• Being a queue, we dequeue the first entry in it using shift operator on currentQueue.
• Now, we pass node(Node(3)) and isNextLevelEven inside findNextNodes function whose purpose it to return us a list of children of the passed node according to isNextLevelEven boolean variable.
• findNextNodes like before initializes an empty nextNodes array and pushes the left and right children of node if they exist. The change comes in the returning value step where now exists a ternary condition where isNextLevelEven if true will return the reversed nextNodes (for right to left) array else the nextNodes (for left to right) as it is . In this step, since isNextLevelEven is true, the returned value will be [Node(20),Node(9)].
• Then the returned array is concatenated with nextNodes. nextNodes will be [Node(20),Node(9)].
• Now before the end of the loop, if currentQueue is empty and nextNodes is not, currentQueue starts referring the nextNodes array and nextNodes is reinitialized to []. We also flip the isNextLevelEven boolean. Also, all the node.val values in the currentQueue are pushed in the form of an array to output. By this step, output will be [[3],[20,9]].
• The algorithm seems reasonable till here. But let's see what happens in next iteration of while loop.
• Now currentQueue being [Node(20),Node(9)], node will be Node(20) due to shift operation.
• Now since next level is 3 which is odd, [Node(15),Node(7)] will be concatenated with nextNodes.
• And again in next iteration of while loop, node will be Node(9) due to shift operation resulting in nextNodes being [Node(15),Node(7),Node(23),Node(43)].
• Now when the if condition inside the while loop holds true, output will become [[3],[20,9],[15,7,23,43]] which clearly is wrong because the last level is not traversed from left to right as we wanted to.

The step where we went wrong here is assuming that currentQueue well has to be a queue. Being a queue, we will never be able to start a traversal from the last element. For instance, after we got correct output till level 2, we would have wanted, that for level 3, first the children of Node(9) were pushed from left to right and then for Node(20) but instead the reverse happened. This is literally how I formed the second intuition that it's my last element from which the next level of nodes should be traversed. And the Data Structure which operates primarily on the last element is none other than a stack.

So all we need is to replace the shift operation with pop and things will work as we intend to. Obviously, it's better to rename currentQueue to currentStack for brevity.

So here is the working code :-

var zigzagLevelOrder = function(root) {
    if(!root)
        return [];
    let currentStack = [];
    let nextNodes = [];
    let isNextLevelEven = true;
    const output = [[root.val]];
    currentStack.push(root);
    while(currentStack.length){
        const node = currentStack.pop();
        nextNodes = nextNodes.concat(findNextNodes(node,isNextLevelEven));
        if(!currentStack.length && nextNodes.length){
            currentStack = nextNodes;
            nextNodes = [];
            isNextLevelEven = !isNextLevelEven;
            output.push(currentStack.map(node=>node.val));
        }
    }
    return output;
};

const findNextNodes=(node,isNextLevelEven)=>{
    const nextNodes = [];
    if(node.left){
        nextNodes.push(node.left);
    }
    if(node.right){
        nextNodes.push(node.right);
    }
    return isNextLevelEven?nextNodes.reverse():nextNodes;
}

The second question was asked to me in a startup's interview in early stage of my career. That time I didn't know JS and was interviewing for a Java Dev position. I didn't do well in that interview and kind of dreaded this problem. Later one day while solving the level order traversal on leetcode, I realised that the zigzag problem is an extension of this and then implemented it in same manner as I documented in this article. That connection between both problems was cool to arrive at.

Thank you for your time :D

The number of unique methods that existed within the Class responsible for this were 30. Now for urgency sake, I went berserk and added 30 console.log inside each such method to log the method names.

But obviously, I didn't like it much. There has to be some way to achieve the same with some extra fancy code.

That's where I begin my intense google search to end up with 2 implementations.

But before that, let's consider a simplified version of my problem :-

Class OperationManager {

operation1(){}

operation2(){}

operation3(){}

operation4(){}

operation5(){}

operation6(){}

operation7(){}

operation8(){}

operation9(){}

operation10(){}

}

function managerFactory() {
  return new OperationManager();
}

//Assume usage inside another function like so :-
(function(manager) {
  manager.operation8();

  manager.operation1();

  manager.operation4();

})(managerFactory());

Now I want to know the name of each operationX method that is called without adding a single line of console.log('operationX called') line inside the function bodies.

Time to check those 2 implementations :-

1. Injecting logging behaviour

function logFnCall(func) { 
  const p = document.createElement('p');
  p.textContent = `${func} called`
  document.body.append(p);
}

function inject(obj, func) {
  for (let key in obj) {
    let value = obj[key];
    if (typeof value === 'function') {
      obj[key] = function() {
        func(key, value);
        return value.apply(this, arguments);
      }
    }
  }
}

// Modify the managerFactory() implementation like so :-
function managerFactory() {
  const manager = new OperationManager();
  inject(manager, logFnCall);
  return manager;
}

Explanation :-

The inject function is trying to loop over all the keys that are available inside the obj and then if the value obtained using that key is a function, we modify the obj[key] to accommodate our custom func implementation and then call the old implementation using appropriate context and arguments. So inside the managerFactory function, instead of returning new instance as it is, we create it and save it in manager and before returning the same, we inject our custom logFnCall implementation.

Here is the codepen for the same :-

Wow so nothing came up. It's totally blank !!!!!

And there is a good reason for that:-

In the inject implementation, we have this statement - let key in obj which assumes that each key in obj is enumerable. But obj was created using a class and class methods are non-enumerable. So we just need to modify this line to help us fetch those method names. We can do it successfully by replacing the existing statement with let key of Object.getOwnPropertyNames(obj.constructor.prototype).

Here is a working codepen :-

2. Proxifying logging behaviour (I personally liked this)

function logFnCall(key,value) { 
  if(typeof value==='function' ){
  const p = document.createElement('p');
  p.textContent = `${key} called`
  document.body.append(p);
  }
}

function proxify(obj, func) {
  return new Proxy(obj, {
    get: function(target, key) {
      let value = target[key];
      func(key, value);
      return value;
      // or return Reflect.get(...arguments) 
    }
  })
}

// Modify the managerFactory() implementation like so :-
function managerFactory() {
  const manager = new OperationManager();
  const proxyManager = proxify(manager, logFnCall);
  return proxyManager;
}

Explanation :-

The proxify function returns a new Proxy object which has a get trap setup. The get trap has target which is obj itself and key which can be any key in the obj itself or it's prototype and so forth. Whenever we do something like manager.operation1, the internal [[Get]] implementation in the JS spec gets called to return us operation1 property on manager. Proxy helps us intercept that call to [[Get]] to perform any operation that we want to. And we have a good use-case here of logging the function name that's called. To do that, we pass a custom func which takes the key and value that's being intercepted and according to inputs performs the relevant operation. We return the value because we want the underlying [[Get]] behavior to remain same. We can also return Reflect.get(...) to get the existing behavior but not using it doesn't pose a problem here. So inside the managerFactory function, instead of returning new instance as it is, we create it and save it in manager and instead of returning the same, we proxify the manager with our custom logFnCall implementation and return the proxyManager.

Here is a working codepen :-

I like the second implementation because of following reasons :-

• No direct mutation of method implementation like in inject.
• The conditional behavior of checking if the value is a function is implemented inside the logFnCall in second approach. Gives more power to the developer to compose custom behaviors.
• Not caring about the enumerable and non-enumerable stuff.

A good resource to know more about Proxy

Have encountered similar situation and have an alternate proposal ? Go smash the comment section then 💻

Thank you for your time :D

But this was a challenge which was organized at the end of a Web Performance Bootcamp held by team devkode (founder - Sunny Puri) on JavaScript Meetup (co-organizer NC Patro) platform. If you're interested in knowing my experience of the same, then here is the link but we will focus more on the challenge here.

Problem Statement :-

When you open and close the amazon.com sidebar, paint flashing occurs which can be eliminated via CSS modification. So you're required to identify those changes and submit the same.

If you're unfamiliar with terms like layout, paint and composition, I would recommend you to go through CRP link to get a know how so you can better relate with the solution ahead.

Alright, so how to first visualize this paint flashing stuff ?

To start with, let's go to amazon.com and there, we will take help of our age old friend DevTools :-

After opening the DevTools, lets do Ctrl/Command + Shift + P to open the Command Menu. Search for Show paint flashing rectangles and select that option.

Now if you try to interact with the left hamburger icon used to open/close the sidebar, you will see green flashing rectangles on the screen indicating paint pipeline is triggered. And it is this paint, we aim to eliminate.

Alright so now we visualize the problem and before I go on to explain the solution, you can also try arriving at the solution. Also, I want you to know that it took me 1.5 hours to identify the CSS modifications. You might take less or more time since that's all relative.

Did you try it ?

Yes/No ? (Drop in the comment section)

I identified two main elements that were responsible for this paint flashing :-

• the CSS class lock-position on body tag

• the sidebar element with id of hmenu-container

body.lock-position

Consider the following CSS when sidebar is open :-

body.lock-position {
    overflow: hidden;
}

This is used by Amazon to prevent the scroll of page content when the sidebar is open. Now if we go to css triggers to see the layout, paint and composition cost of the same, here is what we will find :-

Both overflow-x and overflow-y together combined are equivalent to overflow CSS property and as we can see a change in this triggers the layout pipeline making it an expensive operation.

If we remove, the overflow:hidden from body.lock-position, you will notice that while closing the sidebar, there is no more paint flashing but while opening there still is (Obviously, now the when the sidebar is open, a user can scroll through page content but that was not a constraint). We will eliminate the leftover paint flashing next.

The result till now :-

#hmenu-container

Consider the following CSS when sidebar is closed :-

#hmenu-container {
    position: fixed;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    visibility: hidden;
    z-index: 100000;
}

Consider the following CSS then sidebar is open:-

#hmenu-container.hmenu-visible {
    visibility: visible;
}

You will notice that visibility CSS property initially takes a value of hidden and on click of hamburger icon changes to visible. Now let's go to css triggers to know what part of layout, paint and composition does visibility trigger :-

Now in the bootcamp, there was a certain section where it was talked about how transform CSS property is used to perform UI changes on a separate layer via the compositor thread and will-change CSS property let's the browser know beforehand how an element is expected to change so it can promote the element to a separate layer as an optimization.

Let's look at how transform works from css triggers:-

Cool lets use the above for the final piece of our solution :-

#hmenu-container {
    position: fixed;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    will-change:transform;
    transform: translateX(-100%);
    z-index: 100000;
}

#hmenu-container.hmenu-visible {
    transform: translateX(0);
}

To mimic sidebar being closed, we can add transform: translateX(-100%); to #hmenu-container which will shift the whole container to left by 100% of it's width.

And to mimic sidebar being open, we can add transform: translateX(0); to #hmenu-container.hmenu-visible which will show the whole container in current viewport.

Here is the final result :-

Note:- The above will work even without will-change:transform since that's an additional optimization of promoting #hmenu-container to another layer before the sidebar is made to open.

And that's it for the solution. This challenge was fun because it integrated what we learned at the bootcamp. Also I think the other participants who completed the challenge might have had other approaches. So if you figure out some other way, do share the same in the comments 👇.

P.S. - Btw if you're wondering about those slick looking keyboard keys support in markdown, check out this. Also the cover image is generated using this awesome web-app by Savio Martin.

Thank you for your time :D

Here are two ways which can come into mind :-

const handleClick = () =>{console.log('You can remove me later safely')}
element.addEventListener('click',handleClick);
element.addEventListener('click',()=>console.log('Try and remove me noob'));

Now when it comes to removing these event listeners, it's not possible to remove the second one since it's anonymous and for first one we can just do element.removeEventListener('click',handleClick);

What if I told you there is a way and a syntax you might not be familiar with when it comes to event listeners ?

Well here it is :-

const someObj = {
handleEvent: (e)=>console.log(`I am ${e.type} event`);
}

element.addEventListener('click',someObj);

And :-

Jokes aside, it's always been there. It's just less spoken about. And I came across this when I solved this StackOverflow question and my mind was blowwwwnn !!!

Also, You can just remove the event listener like so element.removeEventListener('click',someObj);

After finding this, I thought to myself that what if I make a bare minimum Handler class which can abstract the registration and unregistration bit and work on the same principle ?

And this is how it looks :-

class Handler {
  #element
  #eventMap = {}

  constructor(element, eventMap) {
    this.#element = element
    this.register(eventMap)
  }

  handleEvent(e) {
    this.#eventMap[e.type](e)
  }

  register(eventMap) {
    this.#eventMap = { ...this.#eventMap, ...eventMap }
    Object.keys(this.#eventMap).forEach((event) => {
      this.#element.addEventListener(event, this)
    })
  }

  unregister(event) {
    this.#element.removeEventListener(event, this)
  }

  unregisterAll() {
    Object.keys(this.#eventMap).forEach((event) => {
      this.#element.removeEventListener(event, this)
    })
  }
}

But what made me go for a class implementation ? Well now we know that we can pass an object to add/removeEventListener, we can have a custom Handler class inside which this will point to the object instance and come into use.

Let's look at a usage sample of this code :-

const handler = new Handler(element, {
  click: ()=>console.log('Yo I am clicky'),
  focus: ()=>console.log('FOCUS!!!'),
});

What the above does it that for element, it registers both the anonymous functions for respective events. And if you go further to register another function for click like so :-

  handler.register({
    click: () => console.log('Well I am new clicky')
  });

This will override the existing click function that we had without any worry of handling its removal and add this new anonymous function.

Now if you want to explicitly unregister the click function, how would you do so ?

handler.unregister('click');

So anonymous or non-anonymous, the Handler class will ensure that for each event type, only one function is registered for the same element. But what if I want to register multiple functions for same event type for the same element ?

Well in that case, you can create another instance of Handler class with same element and let it be responsible for it.

It's still a new concept to me and maybe I might have derived some wrong conclusions. But I will be more than happy to know more about it. Did you know this ? If so, have you used this ? Do you not prefer it ? Any bottlenecks ? Feel free to bash that comment section 💪.

You can go through this article for more insights into handleEvent.

Here is a codepen where you can play with this implementation :-

Thank you for your time :D

The repeat() method constructs and returns a new string which contains the specified number of copies of the string on which it was called, concatenated together.

Now one might think that there is a really straightforward to implement this. Yes there is but if asked in an interview and you go with the straightforward way, they will be like :-

How do I know this ?

Because I got mehhhhd......

So that's why we are going to see few approaches to solve it. The real optimized approach was not intuitive to me and is still something I am trying to wrap my head around. But I came up with a middle-ground approach that works better than the meh!! one.

And once, again we will take a synonym for repeat. Google time !!!!

replicate sounds cool.

Alright let's go implement String.prototype.replicate now :-

The Meh Approach

String.prototype.replicate = function(count) {
  let input = this;
  let result = "";
  for (let index = 0; index < count; index++) {
    result += input;
  }
  return result;
}

Meh explanation :- We initialize result to "" and start a for loop in which we iterate till count and simply keep appending the input to the result variable. Very straightforward but meh!!.

Meh benchmark :-

100 % slower with 108 operations per second compared to 9202566.4 operations per second . Let me cry in the corner.

The Little Less Meh Approach

String.prototype.replicate = function(count) {
  let input = this;
  let result = this.valueOf();
  for (var index = 2; index < count; index*=2) {
    result += result;
  }
  let remainingCount = count - index/2;
  return remainingCount > 0 ? result + input.replicate(remainingCount) : result;
}

Little Less Meh explanation :-

• Let's consider the case of 'hey'.replicate(10) :-
  • We have input initialized to this and result initialized to this.valueOf(). The valueOf() bit helps in decreasing the implicit conversion time that's happening whenever later result will be concatenated to itself.
  • Now the for loop stuff -
    • index is intialized to 2.
    • index should be less than count
    • index should be multiplied each time by 2
    • result will be appended to itself each time in the iteration:-
      • result for index = 2 will become heyhey
      • result for index = 4 will become heyheyheyhey
      • result for index = 8 will become heyheyheyheyheyheyheyhey
      • index will become 16 which is greater than 10 and we exit the loop.
    • remainingCount will be 10 - 16/2 = 2;
    • When remainingCount will be greater than 0, we will recurse by calling input.replicate(remainingCount) and add its result to current result or simply return result.

Little Less Meh benchmark :-

76.79% slower with 2109699.5 operations per second compared to 9091332.85 operations per second . That's still relatively slower than the native one but way way way faster than what we had initially.

Earlier performing the repetitions itself was O(count) but now the same is somewhere down the line of O(log(x)+log(y) +....+log(k)) but not completely O(log(count)).

In 'hey'.replicate(10) scenario :-

• First time, O(log(8)) work is done and then in next recursive step O(log(2)) i.e O(log(8) + log(2)). And if I am doing maths correct,

log(a) + log(b) = log(ab)

That means O(log(8) + log(2)) is O(log(16)) which is greater than O(log(10))(the optimal solution).

The legendary optimal solution I would have never landed upon without the internet

String.prototype.replicate = function(count) {
    let result = ''
    let pattern = this.valueOf();
    while (count > 0) {
        if (count & 1) 
            result += pattern;
        count >>= 1
        if (count) pattern += pattern;
    }
    return result;
};

Noob explanation :- I am still trying to understand the intuition behind this solution but I think it's to do with the fact that every number can be represented in a binary form. So let's say count is 5 then it can be represented as 101 in binary. So it's possible for us to repeat the string count times by just resorting to binary calculations. If we try to differentiate between 4 and 5, we know there is an extra 1 in latter case. Now instead of seeing the above code as some binary work of art, replace count&1 by count%2!==0 and count>>=1 by count=Math.floor(count/2). What this means is that, whenever count is odd, we would want to save the pattern till now in result variable. What is pattern ? pattern is repeated concatenation of itself similar to our earlier algorithm so it will always repeat in powers of 2. It's necessary to take care of the situation when count is not divisible by 2 and store the current pattern in result as we go until count becomes 0.

Did you expect a better explanation ? I can't give it right now since I am a noob in binary land. But maybe somewhere in a parallel universe I invented this Algo and helped Brendan Eich get rid of typeof null -> object 🤷‍♂️.

Best benchmark yet :-

Still 29% slower ? WTH. But hey, I ain't competing with JavaScript engines here .

The Bonus MDN polyfill

 String.prototype.replicate = function(count) {
    var str = '' + this;
    count = +count;
    count = Math.floor(count);
    if (str.length == 0 || count == 0)
      return '';
    var maxCount = str.length * count;
    count = Math.floor(Math.log(count) / Math.log(2));
    while (count) {
       str += str;
       count--;
    }
    str += str.substring(0, maxCount - str.length);
    return str;
  }

Expected an explanation ? I don't care and you will see why 👇

The mandatory benchmark :-

99.94 % slower with 5211.6 operations per second compared to 8344361.29 operations per second . And there is definite reason why it is even slower than what I came up with. What I think is happening is that upto a power of 2 which is less than count, we use the same ideology as in the optimal solution for concatenating and doubling length of str every time. But after that for the remaining length, it uses substring and appends that to str again. It's the second step of substring which makes it a costly operation. Though it does better than the initial Meh solution of 108 ops/s, it's still no where near around the best optimal solution I found online or even mine 😎.

MDN : 0 Lakshya : 1

JK. The site is and hopefully remains a gold mine ❤️.

Here are the overall benchmarks :-

Have something to add on ? Feel free to

Thank you for your time :D

The Node.contains() method returns a Boolean value indicating whether a node is a descendant of a given node, i.e. the node itself, one of its direct children (childNodes), one of the children's direct children, and so on.

But wait, Node.prototype.contains(...) already exists. I want another name for our custom function. Let's google synonym of contains coz

Intense googling later......

Certainly we ain't going with swallow. I think includes would be cool as both Array and String have includes as well in their prototypes.

Before we proceed one important thing to know is that when adding new method to prototype and expecting to use it like so :-

document.includes(document.body),

the method should not be an arrow function so that document can be accessed inside the includes function via this keyword.

Alright then, let's implement Node.prototype.includes in 4 different ways :-

The recursive DFS

1 Node.prototype.includes = function(node){
2 const currentNode = this;
3  if(!currentNode)
4   return false;
5  if(currentNode===node)
6   return true;
7  let isNodeFound = false;
8 for(let index = 0;index<currentNode.childNodes.length;index++){
9    isNodeFound = isNodeFound || currentNode.childNodes[index].includes(node);
10   if(isNodeFound) return true;
11  }
12  return false;
13 }

Explanation :-

• Line 2 to 4 - Set currentNode to this and If currentNode doesn't exist, simply return false.
• Line 5 to 6 - if currentNode is equal to node return true.
• Line 7 to 13 - Initialize isNodeFound to false. Then loop over childNodes of the currentNode and on each child, call the includes method again to check if they include the node element. If they do, isNodeFound will ultimately become true since it is being Orrrrrrd with the results coming from respective childNodes and reassigned to itself. Once isNodeFound is true, we don't need to loop over rest of the childNodes of currentNode and exit early by returning true else ultimately return false.

The iterative BFS

1 Node.prototype.includes = function (node) {
2 const queue = [];
3  let currentNode = this;
4  queue.push(currentNode);
5  while (queue.length) {
6    currentNode = queue.shift();
7    if (currentNode === node) return true;
8    if (currentNode.hasChildNodes()) {
9      queue.push(...currentNode.childNodes);
10    }
11 }
12  return false;
13 };

Explanation :-

• Line 2 to 4 - Initialize an empty list as queue. Set currentNode to this and push (or enqueue to be specific) it.
• Line 5 to 12 - While the queue is not empty, dequeue the currentNode from front of the queue (using shift here). If currentNode is equal to node then return true. Otherwise enqueue the childNodes of currentNode (using push here). Once we are out of the while loop, we have traversed all the nodes and can safely say that we couldn't find the node and return false.

Note - The above can be transformed to iterative DFS by using pop instead of shift and obviously for the sake of consistency, rename queue to stack.

Till now both the approaches followed the classic DS/Algo traversal with DFS and BFS. We are now going to see 2 more approaches which take benefit of certain properties that are specifically applicable to DOM nodes.

LCRS (Left Child Right Sibling) form

1 Node.prototype.includes = function (node) {
2 const currentNode = this;
3 if (!currentNode)
4   return false;
5 if (currentNode === node) return true;
6 return !!(currentNode.firstChild?.includes(node) || currentNode.nextSibling?.includes(node))
7 };

Explanation :-

• Line 2 to 5 -
  • Initialize currentNode to this and if currentNode doesn't exist, return false.
  • If currentNode is equal to node return true
• Line 6 - DOM nodes not only have pointers to their childNodes but also to their sibling nodes as well as parent nodes. Here we are going to leverage the sibling factor for easy traversal. So, we can now check whether the current node's firstChild includes the node OR current node's nextSibling includes the node. Also notice the !!. That's because I have used the ? operator due to which we can end up with undefined || undefined condition or false || undefined condition where both evaluate to undefined which is a falsy value and so !! will ensure undefined coerces to false.

Using parentNode

1 Node.prototype.includes = function(node){
2 const currentNode = this;
3  while(node){
4    if(currentNode===node) return true;
5    node = node.parentNode;
6  }
7  return false;
8 }

Explanation :-

• Line 2 to 7 - Remember DOM node being so attached to it's siblings and parent ? The latter one works well for this use-case as well. While node exists, we check if currentNode is equal to node and if it is we return true, else the node is made to point to it's parentNode for further comparisons. If we exit the while loop, it's safe to say that the node isn't a contained within the currentNode and thus, return false.

And here is a working codepen with all 4 implementations. Comment the rest for any one to reflect ✨.

Have more ways to implement the same ? Feel free to share your approach in the comment section 👇.

Thank you for your time :D

This is one those questions that I always practice before an interview.

The leetcode problem statement goes like this :-

Given the root of a binary tree, determine if it is a valid binary search tree (BST). A valid BST is defined as follows:

• The left subtree of a node contains only nodes with keys less than the node's key.

• The right subtree of a node contains only nodes with keys greater than the node's key.

• Both the left and right subtrees must also be binary search trees.

There are 3 implementations that I know which can help us validate a BST.

Inorder traversal with extra space

One of the clean features of a BST is that if you do an inorder traversal of the same, you get the node values in a sorted order.

function isValidBST(root){
 const arr = [];
 helper(root,arr);
 for(let index = 0;index<arr.length-1;index++){
        if(arr[index+1]<=arr[index]){
            return false;
        }
  }
    return true;
}

function helper(root,arr){
    if(!root)
        return;
    helper(root.left,arr);
    arr.push(root.val);
    helper(root.right,arr);
}

Approach breakdown :-

1. Initialize an empty array arr.

2. Call helper(root,arr) which internally does :-

   1. Traverse the BST in inorder fashion.

   2. Push each root.val inside the arr.

3. Then we loop over the arr and for any index if an element is less than or equal to previous element, then we simply return false. This is because elements should have been strictly increasing as per the requirements.

4. Otherwise, we return true.

Inorder traversal without extra space

It's possible to do the above and exit early if there is an invalid BST without using extra arr space.

var isValidBST = function(root){
    const prev = helper(root,null);
    return prev.isNotValid ? false : true;
    }

function helper(root,prev){
    if(!root)
        return prev;
    prev = helper(root.left,prev);
    if(prev && root.val <= prev.val){
        prev.isNotValid = true;
    }
    if(prev?.isNotValid)
       return prev;
    prev = root;
    prev = helper(root.right,prev);
    return prev;
}

Approach breakdown :-

1. Let's consider helper(root,prev) first (prev represents previous node) :-

   1. if(!root) return prev - If the root is undefined , we return the prev element.

   2. prev = helper(root.left,prev) - We will first go through the left subtree for each root to find the prev element.

   3. if(prev && root.val <= prev.val){ prev.isNotValid = true; } - Once we return from the left subtree , if prev exists, we compare root.val and prev.val to check if current root.val is less than or equal to prev.val. If it is, we create a property on prev by the name of isNotValid and set it to true.

   4. if(prev?.isNotValid) return prev; - Next we check if this prev.isNotValid exists or not and if it does then we simply return prev to exit early and not further proceed for subsequent right subtree.

   5. prev = root - This is how we set the prev value to root so that for next node we can use this prev value for necessary comparisons.

   6. prev = helper(root.right,prev); - Going through the right subtree for each root to find the prev element.

   7. return prev; - It's essential to return the prev to the calling function for value to reflect.

2. const prev = helper(root,null); - Inside isValidBST, we get the prev element from helper(root,null).

3. return prev.isNotValid ? false : true; - If prev.isNotValid exists then that means the BST is invalid and we return false else we return true.

Utilizing the BST property

In BST we can say that each node value will be more than it's left ancestor and less than it's right ancestor for it to be valid. This is what we are going to use now :-

var isValidBST = function(root){
       return helper(root,-Infinity,Infinity);
   }
function helper(root,leftMax,rightMax){
    if(!root)
        return true;
    if(root.val > leftMax && root.val < rightMax) {
        return helper(root.left,leftMax,root.val) && helper(root.right,root.val,rightMax);
    }
    return false;
}

Approach breakdown :-

1. Let's consider helper(root,prev):-

   1. if(!root) return true; - If the root is undefined we can say that the BST is valid till now.

   2. if(root.val > leftMax && root.val < rightMax) { return helper(root.left,leftMax,root.val) && helper(root.right,root.val,rightMax); } - This is the core logic where we compare root.val with leftMax and rightMax. Only if root.val is greater than leftMax and root.val is less than rightMax, we can proceed further to check for corresponding left subtree and right subtree and it's required that both of the subtrees need to return true for the BST to be valid. In case of left subtree, rightMax will change to current root.val and in case of right subtree, leftMax will change to current root.val.

   3. If the above condition fails, then we know it's not further required to check for any subsequent left or right subtree and simply return false.

2. Inside isValidBST, we do return helper(root,-Infinity,Infinity); and pass leftMax as -Infinity and rightMax as Infinity as initial values for our root node.

Out of all the approaches the last one is really clean and I guess an interviewer might expect it. I have given interviews where the first approach was enough and interviewer didn't ask for any optimizations. But if they do, I might skip the second one and jump straight to the third one.

Also I have ignored the space taken by call stack due to recursion and well you never know I might update this article in the future with more approaches if i feel so

Thank you for your time :D