Baby Yoda, Our First Experimental Agent

Introduction

Context

There's a saying - necessity is the mother of invention. This is certainly the case with Baby Yoda, an AI agent I've customized due to the demands of my work as a software engineer.

First, let me shed light on the reasons why AI, for me, is such a necessary tool. Not only as a Software Engineer but also as an individual.

These are the reasons the current stage of AI helps me immensely:

First, it kicks off the first step. What I mean by that is that it's a great tool for 'coming up with solutions'. Sometimes, I don't really know how I want to approach a certain problem. Even if I have a certain idea of how to do so, I can quickly create many scenarios, analyze their pros and cons, and THEN proceed the 'engineering' way. Coming up with scenarios, if you have the 'engineering mindset' into the prompts, allows you to iterate faster between the solutions you previously thought about. And that's invaluable to me. Not just in my work, but I'd say in life too.

Let's say I want to create a piece of system I have in mind, AI allows me to iterate between different ideas I have. It's an extremely efficient creative tool.

I'm on the team that thinks that you should never let the AI do your work. But if you are an extremely practical person, go for it, knowing that it will come with costs.

That being said, my biggest usages for AI are:

• Brainstorming solutions.
• Learning concepts with the use of trusted sources. (I'll explain this in a later post)
• Automatizing things.

After working with the AI environment for about 2+ years now, let me share some of my pain points and what I did to find a way to overcome them.

Pain Points

In short, my primary concerns while working with existing AI tools were:

Frequent loss of data

Since I was trying to understand how to work with LLMs, trialing and making errors was a big part, and with that, the loss of data was significant. Important prompts were lost or forgotten, constant copying and pasting. And limitations over the context make this also a hassle.

Difficulty in adjusting contexts within conversations

Another big pain point when working with LLMs is the context window you are allowed to work with. Especially, ensuring that the information in the context is valuable. This is probably what separates a great AI user from a mediocre one because you have to understand how the underlying technology works. And providing more data often means broader answers. For example, when people 'cheat' to get forbidden answers from the AI, what they usually do is provide a larger context window. When the models are fine-tuned/trained to reject a certain type of answer, you can change the probabilistic 'distance' of it providing more context thus 'distracting' the AI that it is a 'forbidden' answer. Same things goes for any other answer that you actually want. Sometimes providing more context means less accuracy.

Hassle of utilizing the conventional web interface

If you are a professional, you are probably used to your own workflow. And if you are a programmer, that's your editor. So having a web interface + chat is just not great.

Inability to expand and customize the system

As a programmer, having the freedom to tweak and expand the tools you work with is essential, especially with this great analytical tool that is GPT-4. So, this would pretty much make every other tool obsolete, as I couldn't have the freedom to expand it based on my vision and my own pain points. Having my own system opened up a new avenue for automation and custom commands that I was not able to do before. I explored this to the fullest. More on that later.

Enter Baby Yoda

The Journey

As anyone who has been following me on my Youtube channel (Yes, I know, I need to update it often! 😅) will probably have seen the first initial steps of the chat bot I've been developing inside VS Code. It has stalled for a while due to the preparations for blogs, work, and so on. But I have been working on it for a while now, trying to get interesting ideas, based on these pain points I had. Surprisingly, it has become a daily tool for me. The only web-based tool I sometimes use is the code interpreter to create graphs and the like. Although a simpler version of it can also be created, as I did with the '.dot' language integration to create graphs.

What it proposes

Having all these factors in mind, the Baby Yoda Extension on VS Code is a system that allows me (and every other person really, VSCode is a great writing tool) to:

• Save any meaningful data quickly.
• Quickly create and adjust context based on the workspaces I'm in.
• Tightly integrate into my workflow.
• Expand and customize the system based on my needs.
• Automatize things, fast.

Where we are now

• A fully functional chat bot, that has similar functionalities to Copilot X.

  • It's not aimed at development 'per se'. It's not going to create PR's or auto-completion, but rather tasks that you do every day. For instance: Let's say you are currently learning 'x' topic. You can create an agent for that and then handpick and model the agent.

• Automation. In order to achieve this, we need to be able to expand the server.

  • In a Server that you can expand, and create your own commands and write the set of scripts. The agent can loosely decide which to execute if it's a broader task, or you may constrain a set of them.
    For example: Let's say you want a certain set of actions to be executed by the day and reported back to you afterwards. You would necessarily need to write those set of scripts/functions, but the agent and you can decide how loosely they will act upon those.

This direction, an extremely exciting one, allows us to work towards the concept of assistants that can be quickly generated.

The Challenges

• As a Professional Software Engineer, I still have my own work responsibilities, along with the challenge of not moving as fast as I would like. That means we still have issues to fix while trying to gather people around the project :)

• Also, creating a system that would be useful for not just me but for all professionals comes with its own set of challenges too.

As I don't have any monetary intentions with this project, my goal is to release it into the wild once it's acceptably ready. I have been working towards the setup, which can be a bit painful since you will have to own your own server while using the extension on VS Code.

The Future

• Now, a big part of where I am at the moment is to make everything well-rounded, especially the automation part as the chat bot works fairly well.
• Working on the setup and onboarding process.
• Working on the documentation and the tutorials.
• Spreading the word.
• Trying to captivate other engineers to work with me on this. It's not an easy feat.

Conclusion

I hope you have enjoyed this project. Very soon, I will be releasing a video on my YouTube channel showcasing how beautiful this little program I'm currently using is.

I hope you found this post informative and insightful about the AI space in general. If you are an engineer and have some spare time to help on this project, please don't hesitate to send me an email.

Have a great day!

Matt