fix(lexer): handle non-code characters within language identifier #101

The lexer now correctly handles non-code characters within a language identifier, ensuring that the parsing process is not disrupted. This change ensures that the lexer can distinguish between the start of a code block and non-code content, thereby improving the accuracy of the lexing process.

Author: phodal

exts/devin-lang/src/grammar/DevInLexer.flex

@@ -99,7 +99,6 @@ NEWLINE= \n | \r | \r\n
   {CODE_CONTENT}       { if(isCodeStart) { return codeContent(); } else { yybegin(YYINITIAL); yypushback(yylength()); } }
   {NEWLINE}            { return NEWLINE; }
   <<EOF>>              { isCodeStart = false; yybegin(YYINITIAL); yypushback(yylength()); }
-  [^]                  { isCodeStart = false; yybegin(YYINITIAL); yypushback(yylength()); }
 }
 
 <LANG_ID> {