Menu

GitHub CoPilot is a new tool that uses artificial intelligence to help developers write code faster and better. It is powered by OpenAI Codex, a large-scale neural network that has been trained on billions of lines of public code. GitHub CoPilot can generate code snippets, tests, functions, and entire classes based on natural language descriptions or comments. 

I have been curious about GitHub CoPilot ever since it was announced in 2021, but I was also skeptical about how well it could handle the complex and dynamic nature of software development. We started testing it around the middle of 2023 and had a mix of developer skill level (jr, int, expert) use it on multiple client projects to see how useful it was.  Note that our typical development environment is .NET/Angular/C#/SQL.  

When we first started looking at GitHub Copilot in 2021, we found it added some additional suggestions and would fill in code from comment prompts, but the comments had to be very specific and the amount of code it generated could be less than the prompt. Several of the example videos show Copilot creating large functions of working code from a single comment, however that is because there are samples found in open-source projects. If you were to turn off the feature to allow use of open-source code many of these examples would fail to work. Because of that we decided to wait to see how GitHub Copilot developed. 

In mid 2023 we took another look at GitHub Copilot, and enabled the chat prompt feature, which was introduced after our original assessment. The new chat feature is more effective and helps write a lot of the boilerplate code. This feature is what we were waiting for to make Copilot worth the investment. 

Below are some of the features we liked and disliked about GitHub CoPilot after using it for several months on different client projects: 

What we liked 

Auto-creation of test cases from selected code. We had a well-documented API endpoint and gave Copilot a chat prompt requesting that it create unit tests for all possible results of a given action. Copilot created a new file, using industry standard libraries and created tests that followed the Arrange, Act, Assert method of testing. These tests looked very clean and would be usable with minimal modification. A co-worker used a different mocking library, and we re-ran the same prompt, but added an instruction to use specific libraries and it created new tests using the libraries requested. 

Auto-creation of a service (handler) from a chat prompt. This feature was helpful when I needed to create a new service or handler for a specific functionality or request. I simply had to type a chat-like message describing what I wanted the service to do, and GitHub CoPilot would generate the code for me.  

This feature was handy for creating boilerplate code or scaffolding for new services that I could then customize or refine as needed. 

Inserting code from comment prompts. This feature was useful when I wanted to insert a specific code snippet or library function that I was not familiar with or could not remember. I had to type a comment with a natural language description of what I wanted, and GitHub CoPilot would insert the code for me.  

This feature was convenient for finding and using common or standard code snippets or functions without having to look them up or memorize them. 

Answering questions from comment prompts. This feature was helpful when I had a question or doubt about a certain aspect of the code or the language I was working on. I had to type a comment with a question and GitHub CoPilot would try to answer it for me.  

This feature was handy for learning new concepts or clarifying existing ones without having to search online or consult a documentation. 

Wildly enhanced auto complete. This feature was beneficial when I wanted to write code faster and more efficiently. GitHub CoPilot would suggest relevant and accurate code completions based on the context and the intent of the code. It would also adapt to my coding style and preferences and learn from my feedback. This feature was incredible for writing code with less effort and more confidence. 

Learned from the code I wrote (or changed after it wrote it) and improved next time. This feature was impressive when I wanted to improve the quality and consistency of the code GitHub CoPilot generated. GitHub CoPilot would learn from my edits and corrections and adjust its suggestions accordingly. It would also remember the patterns and conventions I used and apply them to future code generation. For example, if I changed a variable name or a formatting style in one place, GitHub CoPilot would use the same name or style in other places. 

What We Disliked 

The quality of the code it generates varies from one time to the next and I can’t tell why. This feature was frustrating when I wanted to rely on GitHub CoPilot to produce consistent and reliable code. Sometimes GitHub CoPilot would generate flawless and elegant code that worked perfectly. Other times, GitHub CoPilot would generate buggy and messy code that needed debugging and refactoring. I could not figure out what caused these variations and how to avoid them. For example, GitHub CoPilot would sometimes generate code that followed the best practices and standards of the language and framework. Other times GitHub CoPilot would generate code that violated the rules and conventions of the language and framework. 

Copilot can struggle with the implementation for specific libraries. We were investigating the use of NSwag instead of Swashbuckle as NSwag has the option to also create ReDoc style documentation as an endpoint. Trying to get Copilot to generate code to initialize the NSwag library was not a great experience, and we can only assume that’s due to NSwag being less popular. 

It seems to not have the latest features as they come out. I’m guessing this is because the features weren’t available when the AI was trained, and the techniques aren’t widely used in public repositories. This feature was disappointing when I wanted to use GitHub CoPilot to explore and experiment with the latest and most advanced features and techniques of the language and the framework. For example, GitHub CoPilot would not generate code that used primary constructors in C#, even when the prompt contained instructions to do so. 

Conclusion 

GitHub CoPilot is a powerful and innovative tool that can enhance and augment the development process and the developer experience. It can help developers write code faster, better, and easier by generating code snippets, tests, functions, and classes based on natural language descriptions or comments. It can also help developers learn new concepts, clarify doubts, find solutions, and discover new possibilities by answering questions and suggesting code completions based on the context and intent of the code. GitHub CoPilot can also adapt to the developer’s style and preferences and learn from their feedback and corrections. 

However, GitHub CoPilot is not a perfect or a complete tool that can replace or automate the development process and the developer experience. It has some limitations and drawbacks that can affect the quality and reliability of the code it generates. It can also introduce risks and challenges that can affect the security and privacy of the code it accesses. Therefore, developers should use GitHub CoPilot with caution and care, and always review and test the code it produces. 

The $10 per month per developer option sends out telemetry so be aware of that if you are working on software covered under an NDA.  For $19 per seat, you can get a business account for GitHub CoPilot that doesn’t send telemetry. This is the version we will be using at Imaginet. From an ROI perspective, even saving an hour a month of coding time per developer will more than pay for the $19 per month.  

At Imaginet, our developers are always on top of the latest technology trends and updates. Make sure to subscribe to our blog for more helpful tips and tricks and if you are looking to update your applications, fill out the form below. Someone will be in touch.  

Discover More