Lexical Analysis & it's Working | Compiler Design

Lexical Analysis is the first phase of compiler designing. Lexical Analysis helps in converting the source code into a series of tokens. In this video, we will be covering what is lexical analyser & it's working as well.

A lexical analyzer contains a tokenizer/ scanner. In case the lexical analyzer detects any invalid token, it generates an error. The primary role of the Lexical Analyzer is scanning character streams from source code, checking for valid tokens, and then pass data to the syntax analysis phase.

Tokens are sequence of characters which represents any information unit in source code. Lexemes are sequence of the characters in a token. They are generally known as instance of token. There are predefined set of rules, considering which we can say that any lexeme is valid or not. These rules of identifying the validity of lexemes/ tokens are defined by the rules of grammar, with respect to pattern. Pattern describes what can be valid token, and such patterns are defined with the help of regex (regular expressions). There are different types of tokens, which are as follows: Keywords, Special Characters, Identifiers, Strings, Constant, Operators, etc.

Working of Lexical Analyser: The primary task of the lexical analysis is reading input characters in code & producing tokens. Lexical analyzer scans complete source code. It is identifying each token in order. Scanners are implemented in such a way that they produce the tokens only when it is being requested by parser. The command: “Get next token” is sent from parser to lexical analyzer, when it wants the next token in sequence. On acceptance of this command, lexical analyzer then scans input for next token and sends it to the parser.

Roles of Lexical Analyser: Tokenization i.e. Divide the source code into valid tokens. Remove white space characters from the source code. Remove all the comments from the source code. Generate error messages along with the specific numbers of rows and columns. Helps in identifying token into symbol table. Helps to expand the macros if found in the source program.

Lexical Analysis in Compiler Design: https://www.geeksforgeeks.org/introduction-of-lexical-analysis/
Working of Lexical Analyser: https://www.geeksforgeeks.org/working-of-lexical-analyzer-in-compiler/