Not a day passes without a well-known CEO boldly declaring that AI will soon replace manual coding or mentioning the startling number of programmers who will lose their jobs to AI in a matter of months.
According to Airbnb’s latest disclosures, teams will probably use AI more and more to manage and move codebases. About 3,500 React component test files were updated from Enzyme to React Testing Library (RTL) as part of the company’s first extensive, LLM-driven code migration.
“We’d originally estimated this would take 1.5 years of engineering time to do by hand, but — using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks,” said Charles Covey-Brandt, a software engineer at Airbnb, in a blog post.
Since Enzyme’s extensive access to component internals no longer matched contemporary React testing procedures, the business sought to abandon it.
75% of Target Files Migrated in just 4 Hrs
The business successfully converted hundreds of enzyme files to RTL in a matter of days utilizing LLMs, a notion that was proven in mid-2023. Last year, the business built a scalable pipeline for a “LLM-driven migration.”
The migration was divided into automated validation and refactoring processes as part of the pipeline. According to Covey-Brandt, “each file moves through stages of validation, and when a check fails, we bring in the LLM to fix it.” The business acknowledged that moving hundreds of files at once was simple with this method.
Airbnb first tried prompt engineering to increase migration success, but ultimately discovered that a brute-force retry cycle worked best. It put in place a system that performs validation many times for every migration stage, dynamically updating the prompts with the most recent file versions and problems.
Airbnb raised the prompt context in addition to the number of repeat tries. LLMs were able to comprehend common testing techniques, team-specific patterns, and the general architecture of the codebase with the use of a context-rich prompt engineering methodology.
“Our prompts had expanded to anywhere between 40,000 to 100,000 tokens, pulling in as many as 50 related files, a whole host of manually written few-shot examples, as well as examples of existing, well-written, passing test files from within the same project,” he said.
In approximately four hours, Airbnb was able to transfer 75% of the target files using the aforementioned methods. 900 files, meanwhile, continued to fail the step-based validation requirements. After then, the business developed tools for target reruns. Each file’s progress was monitored by a migration status remark, and a re-run feature enabled filtering by failure step.
“After running this ‘sample, tune, sweep’ loop for 4 days, we had pushed our completed files from 75% to 97% of the total files, and had just under 100 files remaining,” said Covey-Brandt. Further repeat attempts for the remaining files, according to the business, felt like they were “pushing into the ceiling” of what they could automate. The remaining files were handled by hand.
The success story of Airbnb is not unique to AI-driven code migrations. Similar revelations have already been made by industry titans like Google and Amazon.
Google Saw a 50% Improvement in Speed
Google released a thorough study earlier this year that described their experiences with LLMs for code migrations in a variety of scenarios. It was discovered that migrations triggered by LLM speed up the procedure by 50%.
The business used the Google Ads code base as an example of how to change 32-bit unique ID types to 64-bit ones. Because 32-bit IDs were vulnerable to reaching their maximum value, which might result in integer overflow and system problems, the change was required.
“The full effort, if done manually, was expected to require hundreds of software engineering years and complex cross-team coordination,” Google stated, enumerating a number of possible difficulties throughout the conversion process. The business then developed a process that included a human engineer or expert and a migration toolset based on LLM. According to Google, 20% of the code changes in the change lists were either human-written or amended, with the remaining 80% being entirely AI-authored.
However, the business also stated that because of errors or unnecessary changes, engineers must reverse or modify certain AI-generated updates. “This observation led to further investment in LLM-driven verification to reduce this burden,” said Google.
Do Engineers Like AI-Driven Migration?
Research on the collaboration between humans and AI in code migrations was carried out by Amazon Web Services (AWS). The study concentrated on utilizing Amazon Q Code Transformation to convert outdated Java code to a more recent version. Eleven software developers were interviewed for the research, which found that developers saw AI as a cooperative partner.
According to the survey, developers want to be in charge of the migration process, want to direct the AI according to their knowledge, and act as reviewers to carefully check the modifications.
“Just as code reviews help junior developers improve, constructive user critiques of the AI system enable it to better align with expectations and continuously amend its understanding like a programmer assimilating feedback,” stated AWS in a section of the report.
In order to assist align expectations, the research also recommended that designers of human-AI partnership systems disclose limits rather than hiding flaws.
Several instances of the Java dependent being changed without first confirming that it was the proper version were discovered by the investigation.
“I feel like I am being gaslighted here. The first article says one version, and the second says another version. Which one is it?” said participant 9 (P9), responding to the error.
Additionally, participants indicated that they wanted to double-check everything, even when the AI supplied current output.
“I don’t know if any AI, I would find extremely trustworthy. I am still going to double-check everything. I don’t expect it to be malicious, it gave me a valid dependency update, but I am still going to double-check everything (sic),” said P9.
The code migrations described above are driven by LLMs, although they are not entirely finished by LLMs. Human control, evaluation, and verification are still required.
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream formerly known as CIO News LinkedIn Account | The Mainstream formerly known as CIO News Facebook | The Mainstream formerly known as CIO News Youtube | The Mainstream formerly known as CIO News Twitter |The Mainstream formerly known as CIO News Whatsapp Channel | The Mainstream formerly known as CIO News Instagram
About us:
The Mainstream formerly known as CIO News is a premier platform dedicated to delivering latest news, updates, and insights from the tech industry. With its strong foundation of intellectual property and thought leadership, the platform is well-positioned to stay ahead of the curve and lead conversations about how technology shapes our world. From its early days as CIO News to its rebranding as The Mainstream on November 28, 2024, it has been expanding its global reach, targeting key markets in the Middle East & Africa, ASEAN, the USA, and the UK. The Mainstream is a vision to put technology at the center of every conversation, inspiring professionals and organizations to embrace the future of tech.
