Engineer-4o

Inspiration

As two first year students, we came into our internships excited to build cool projects that directly impact industry. However, instead of learning about CUDA and optimizing algorithms, we learned about AGILE and closing tickets. Programming is involving more and more repetitive and menial tasks which was a major pain point for us since we wanted to spend more time building cool things rather than writing documentation.

What it does

Engineer-4o scrapes Github repositories and issues to gain a global context of what a codebase does. It then searches through the issues and solves them allowing the developers to spend more time tackling the larger overarching issues. Furthermore, many codebases are often not well documented so Engineer-4o analyzes the codebase and can explain to you any questions you have about it so you don't need to waste time tracing variables with inconsistent names across hundreds of lines. This is all packaged in a lightweight VSCode extension that allows you to view github issues and the agent-derived solution in one screen so you don't waste time switching tabs.

How we built it

For the backend of the questioning system, we used GPT-4o due to its lightning-fast response times. Its able to take a large context window and quickly search the vector database for the answer to your question. It can easily cite where it got its information from so you can reference the exact lines of the codebase too! The issue system is run on Gemini Pro. It analyzes the issue and creates a step-by-step process to solving it using Chain-of-Thought prompting. This is then passed to a multi-agent simulation - similar to DEVIN's architecture - which reasons through the debugging process to find the issue in the code. The agents also have tooling to directly modify code and run it so, the agents can modify the code for you and check if it fixes the issue. Then, the Github issue can simply be resolved automatically! To make this as easy to use as possible, we've packaged this software as a VScode extension which easily integrates with your IDE to make development as quickly as possible. We've integrated github so you can sign-in and view issues all from your IDE. This makes development quick and easy - no more switching tabs.

Challenges we ran into

There is limited useful documentation on how to build a complex VS Code extension. So, we had to learn how to build this ourselves through trial-and-error and looking at examples. This took up a huge amount of time but we eventually built out the extension! We also ran into issues using VertexAI because we had assigned the incorrect roles. The agent-based simulation as difficult but reading some of the existing literature on arxiv helped us create a sound architecture.

Accomplishments that we're proud of

We are immensely proud of the seamless integration of multiple advanced technologies to create a powerful tool for developers. Successfully building a fully functional VS Code extension, despite the initial lack of documentation, was a significant achievement. We're also proud of the robust backend system powered by GPT-4o and Gemini Pro, which can autonomously analyze and resolve GitHub issues, streamlining the development process. Our agent-based simulation, inspired by DEVIN's architecture, was another milestone, demonstrating our ability to implement cutting-edge AI techniques. Moreover, the positive feedback from early users has validated our hard work and innovative approach.

What we learned

Throughout this journey, we learned the intricacies of building a VS Code extension from scratch, a skill that required patience and perseverance. We deepened our understanding of advanced AI models like GPT-4o and Gemini Pro, and how to effectively integrate them into a practical application. We also learned the importance of correct role assignments in VertexAI, which was a crucial lesson in managing cloud-based AI services. Additionally, the process of creating a multi-agent simulation for debugging enhanced our knowledge of AI architectures and their practical implementations in software development.

What's next for Engineer-4o

The future of Engineer-4o is bright and full of potential. Our next steps include enhancing the AI's capabilities to handle even more complex issues and expanding its understanding of various programming languages and frameworks. We aim to refine the VS Code extension to offer even more seamless integration and user-friendly features. Additionally, we plan to incorporate more collaborative tools, allowing developers to work together more efficiently within the extension. We're also exploring partnerships with major tech companies to bring Engineer-4o to a wider audience, helping developers everywhere spend more time on innovative projects and less on repetitive tasks.

Built With

• cloud
• javascript
• openai
• python
• react
• vertex