Everything you wanted to know about web tokens, but were afraid to ask

Ekaterina Gainullina, Security Vision

What is a web token?

A web token is a trust token, a digital identifier that the server issues to the user after successful authentication. After receiving the token, the client transmits it with each request, and the server uses this token to verify that the request comes from an authorized user. Usually, a token does not contain secrets like a password. In practice, this is either a random session identifier or structured data with a signature, such as a JSON Web Token (JWT). The digital signature protects the contents of the token from forgery, and the server can verify the authenticity of the token by verifying the signature or by recording an active session. In terms of meaning, the token can be compared to an identity card, which confirms that the verification has already been completed and re-entering the password is not required with each request.

Examples of web tokens:

1. Session ID, or session ID. After logging in, the server creates a session record and sends the client a unique identifier, usually in a cookie. The browser stores a cookie and automatically adds it to each request. The server finds the corresponding session by its ID and thereby recognizes the user.

2. JSON Web Token, or JWT. This is a token in the form of a three-part encoded JSON: header, payload, and signature. The parts are Base64 encoded and separated by dots, for example xxxxx.yyyyy.zzzzz. The payload contains information about the user and access rights. JWT is self-sufficient: the server does not need to store the status of such a session, it is enough to check the signature and expiration dates to make sure that the token has not been changed and is still valid.

3. Access and upgrade tokens in OAuth 2.0. Usually a pair of a short-lived access token and a longer-lived refresh token is used. An access token gives access to resources for a limited time, usually from a few minutes to an hour. The refresh token is stored securely on the client side and is used to receive a new access token after its expiration date. This scheme allows you not to create one long-lived token and at the same time does not force the user to log in frequently.

4. SSO tokens for single sign-on. They are issued by a central identification provider, for example, in the SAML-assertion or JWT format. After successful authentication with the provider, the user receives a token and can access related applications without re-entering the password. The token contains statements about the user's identity and roles, as well as metadata about the time of issue and expiration date. The specific format depends on the single sign-on system: JWT is used in conjunction with OpenID Connect or SAML XML statements.

Why are tokens needed?

Tokens solve several tasks related to user convenience and security, as well as the scalability of web applications.

Their main advantage is convenience, as the user does not need to enter a password every time they access the system. It is enough to authenticate once, after which the token records the fact of login and is used for all subsequent requests. This significantly improves the user experience and eliminates the need to constantly verify your identity at every step.

The second important reason for using tokens is to increase the level of security. A password is valuable and long-lasting information, and sending it repeatedly over the network increases the risk of interception. In an ideal scheme, the password is used only once when logging in, and all further requests are verified using a token. Such a token can be made short-lived and limited in rights. If it is compromised, the damage will be significantly lower than if the password is leaked: access will be limited to a short period of time or a narrow set of actions. In addition, the token can be easily revoked or replaced without user intervention, which cannot be done with a password without changing the usual work scenario. Tokens also reduce the consequences of leaks: even if they are intercepted, the user's password remains secure, and an attacker will not be able to reuse it in other systems.

From an architectural point of view, scalability also plays a special role. Classic sessions require the server or database to store information about all active users, which becomes a problem under heavy loads. JWT and similar solutions contain the necessary data inside themselves, and it is enough for the server to verify the signature. This unloads the infrastructure and allows you to scale the system horizontally: any server instance can process a request without accessing the common session database. This approach is especially useful in distributed architectures and microservices, where tokens can be verified by different systems that trust the same authentication center.

Finally, transmission security is important. Tokens are often used in the Bearer format and are transmitted in HTTP request headers, for example, Authorization: Bearer <token>. In combination with HTTPS, this provides reliable protection when transmitting over the network. Unlike cookies, which are automatically included by the browser in all requests and can cause difficulties with cross-domain interactions, tokens in headers allow you to flexibly manage authorization between different domains and services. However, using a secure connection (TLS) is a prerequisite.

The main types of web tokens

There are several main types of tokens, each with its own pros, cons, and applications:

In real-world projects, a combination of these approaches is often used: for example, single sign-on via SSO issues a JWT, which is then used to authorize API requests, and an Access/Refresh bundle is used inside the mobile application to extend the session. This multi-level scenario allows for simultaneous consideration of security, convenience, and scalability requirements.

How Web tokens work

The process of using web tokens can be described by a sequential chain of actions. First, the user enters their credentials, such as login and password, and requests access to the system. The server verifies the correctness of this data and, if it is correct, considers the user authenticated. After that, the server generates a token: This can be a session record with a unique identifier, a digitally signed JWT, or a bundle of access and refresh tokens under the OAuth protocol. The generated token is sent to the client either as a cookie, in the response body, or in a special header.

After receiving the token, the client application saves it for future use. In the browser, these are most often cookies that will be automatically applied to each request, or local secure storage if JWT is used. Proper storage is critically important because the token is the key to the account.

Each time a protected resource is accessed, the client applies a token. In the case of session id, the browser sends cookies automatically. JWT or access tokens are usually passed in the Authorization: Bearer <token> header. The server receives the request, extracts the token, and verifies its authenticity. In the case of session id, the presence of a valid session in the database is checked, in the case of JWT, the signature and expiration date are checked, and when working with OAuth, an additional request can be made to the authorization server to confirm the validity of the token. If the token passes verification, the server considers the user authenticated and evaluates the access rights that can be specified in the token itself.

Since tokens have a limited lifetime, they cease to be valid after a while. Refresh tokens are used to avoid forcing the user to re-log in too often. When the access token expires, the client accesses a special point on the server and presents the refresh token, receiving in return a new access token, and sometimes an updated refresh token. This process runs in the background and is usually invisible to the user. However, refresh tokens also have an expiration date, after which you must re-authenticate to continue working.

Finally, an important step is to log out correctly. When working with classic sessions, the server deletes the corresponding record, making the token invalid. In the case of JWT, the situation is more complicated, since such tokens are not stored on the server. Blacklisting or token versioning mechanisms are used to revoke them. In OAuth scenarios, the authorization server can mark the refresh token as revoked, which eliminates the possibility of issuing new access tokens. Thus, the process of web tokens covers all stages from the initial issuance to the subsequent update and the correct end of the session.

Rakes and antipatterns

As with any technology, when working with web tokens, there are typical mistakes and practices that should be avoided, as they can significantly weaken the protection of the application.

One of the common problems is storing JWT in localStorage. Despite the convenience of this method for front-end developers, it makes the token vulnerable in the presence of an XSS vulnerability: an attacker can gain access to the contents of the repository and use the token to hijack a user session. Unlike cookies, for which you can set the HttpOnly flag, which restricts access from JavaScript, there is no built-in protection in localStorage. A more reliable approach is to store tokens in secure HttpOnly cookies with the Secure flag, which increases the level of security, although it requires additional configuration of CORS and CSRF protection.

Perpetual tokens are another major mistake. A token with no lifetime actually turns into a permanent pass: if it is compromised, the attacker gets unlimited access. To reduce risks, each token must be assigned an expiration date. Even refresh tokens are usually limited to a few weeks or months. The short lifetime of the access token minimizes the damage caused by its leakage, and timely revocable updates support user convenience. Information security practice recommends using minimally sufficient action intervals and revoking tokens at the slightest suspicion of compromise. The real incident with the leak of the PyPI administrator's personal token in 2024 showed that long-lived tokens pose excessive risks, and confirmed the need for their strict control and automatic expiration.

Excessive saturation of the token with data is also considered an error. If you put a full set of rights, profile, or other voluminous information in the JWT, the token quickly becomes too large. With each request, it is forwarded along with the headers, and HTTP headers have size limits. This reduces efficiency and can lead to technical errors. The optimal approach is to store only the minimum necessary information in the token: the user ID, basic roles, and brief claims. Detailed information should be requested separately, if necessary. In addition, it should be borne in mind that the contents of the JWT are readable by anyone who has the token itself, so the inclusion of confidential information in it is unacceptable.

Finally, one of the critical errors is insecure work with the secrets used to sign tokens. When using symmetric algorithms (for example, HS256), the secret key often turns out to be insufficiently secure: it can be stored in the repository code or included in open configurations. Getting such a key into the hands of intruders means the possibility of creating valid tokens on behalf of the server and completely compromising the authentication system. There have been cases of leaks of JWT secrets into public JavaScript files and integration modules of third-party services. A reliable option is to store secrets only on the server side, use secret managers or environment variables, and rotate them regularly. A more secure solution is to switch to asymmetric algorithms (RS256, ES256), in which the private key is stored exclusively on the server, and only the public key is transmitted to clients to verify the signature.

Thus, attention to the method of storing tokens, their validity periods, the structure and protection of signature keys is a prerequisite for building a secure authentication system.

Myths and reality about tokens

There are several persistent misconceptions related to the use of web tokens.

The first myth is that using JWT supposedly guarantees absolute security. In practice, this is not the case. The token's signature does protect it from forgery, however, if an attacker gets hold of a valid token, he will be able to perform actions on behalf of the user until the expiration date. In addition, the contents of the JWT are not encrypted by default, but only encoded. Anyone with a token can decode it and read the information in the payload. This is normal and complies with the standard, but it imposes restrictions: confidential data cannot be placed in the token. The basic principle of JWT security is not to conceal information, but to make it impossible to imperceptibly replace it without knowing the secret key. Therefore, it is critically important to transfer tokens only over a secure connection, not store them in unsafe places and limit their validity period.

The second misconception is related to the opinion that refresh tokens are not needed and you can limit yourself to one access token. The rejection of the update mechanism leads either to an excessively short access token lifetime, which worsens the user experience, or to its excessive "durability", which reduces the level of security. Using a combination of a "short-lived access token and a longer refresh token" allows you to balance convenience and protection. If the access token is compromised, the attacker will only be able to use it for a limited time, and if the refresh token is compromised, the administrator will be able to quickly revoke it and block further access.

The third common misconception is that tokens can be stored anywhere. The storage location directly affects the system's resistance to attacks. Placing the token in a JavaScript environment (for example, in localStorage) makes it vulnerable to theft through XSS attacks. Storing HttpOnly cookies protects against XSS, but increases the risk of CSRF attacks, since such cookies are sent automatically by the browser. There are standard measures to protect against CSRF, including the cookie's SameSite attribute, the use of anti-CSRF tokens, and verification of Origin headers. The best practice is to store sensitive tokens in HttpOnly and Secure cookies. In cases where cookies cannot be used (for example, in mobile applications), tokens should be stored in secure vaults or in memory, but not transmitted through URLs and other potentially unsafe channels.

Practical examples

The experience of large services confirms the need for proper work with tokens. Google and other service providers use an OAuth 2.0 scheme based on a pair of tokens. The user receives an access token valid for about an hour and a refresh token for its subsequent renewal. As long as the access token is valid, the application can access the API. After the expiration of the access token, it receives a new one by presenting a refresh token. At the same time, the user does not notice the process, and the system retains both security and convenience of operation.

GitHub uses Personal Access Tokens (PAT). These tokens are used to work with APIs and repositories without a password and can have different scopes. PATS are convenient for automation, but when they are compromised, the attacker gets all the rights granted to the token. Therefore, GitHub allows you to limit the validity period, scope, and recommends using "finely tuned" tokens for specific tasks. Practice shows that a PAT leak can have serious consequences: in 2024, the discovery of the PyPI administrator's personal token in a public Docker image posed a threat to the security of critical projects, and only its timely cancellation prevented the incident.

Another typical scenario is token leaks in open repositories. Attackers systematically monitor the appearance of such data on platforms like GitHub. The discovered tokens are often used to access cloud resources or services, which leads to direct financial losses or data leaks. Leaking the secret key for JWT signing makes it possible to issue fake tokens, leaking the refresh token allows access to be extended for a long time, and compromising the session ID via XSS or an unencrypted channel is effectively equivalent to hijacking the session.

Practice confirms that tokens should be handled with the same care as passwords. They should not be stored or published in open sources, transmitted through unsecured channels, or left without expiration date. If a compromise is suspected, it is necessary to immediately revoke tokens and check logs to assess what actions could have been performed using them.

Checklist for safe work with tokens

Below is a short list of recommendations for developers and security experts to help avoid common problems with web tokens:

1. Token storage.
Use secure cookies with HttpOnly and Secure flags for web applications. This excludes access to tokens from JavaScript and guarantees their transmission only over HTTPS. If cookies cannot be used (for example, in mobile applications or third-party REST clients), store tokens in secure storages provided by the platform (Keychain on iOS, Keystore on Android). As a last resort, storage in memory is allowed, but not in long-term open storage. Avoid placing tokens in localStorage due to the risk of XSS.

2. Expiration date and update.
Set a minimum sufficient lifetime for access tokens and use refresh tokens to update them. For example, an access token can be valid for 10-30 minutes, and a refresh token can be updated for up to a day. Ensure that tokens are correctly deleted and disabled when the user logs out. Even refresh tokens should not exist indefinitely.

3. Minimizing rights.
Apply the principle of least privilege. Each token should provide access only to necessary resources and actions. In OAuth, use minimal scopes and prefer Fine-grained tokens to restrict access at the level of specific resources or operations.

4. Audit and review.
Implement a system for logging actions related to the issuance and use of tokens. Log the user's ID, time, device, and key actions. Implement token revocation mechanisms: for sessions, deletion of server records; for JWT, blacklists (according to jti) or rotation of signature keys; for OAuth, centralized revocation of refresh tokens. Provide administrative tools for the forced cancellation of tokens in case of suspected compromise.

5. Token transfer.
Never send tokens to the URL, as they may end up in logs, browser history, or the Referer header. Use headers (for example, Authorization: Bearer <token>) or the body of the POST request. In OAuth scenarios, use one-time authorization codes instead of transferring tokens directly in the redirect parameters. Exclude the distribution of tokens through e-mail, instant messengers or other insecure channels.

By following this checklist, you will close most of the vulnerabilities related to tokens and make life much more difficult for attackers.

Conclusion

Web tokens are an integral part of modern application architecture today. Regardless of whether JWT, OAuth tokens, or session identifiers are used, their main purpose is to provide reliable and convenient confirmation of the user's identity without having to re-enter the password. Tokens make it possible to build scalable and distributed systems, support single sign-on and mobile client operation. This is an effective tool, but it does not solve all problems by itself. The token should be considered as a digital identity card, which must be protected as carefully as a passport or a key card. The loss or compromise of a token means that anyone who gets hold of it can gain access to the system.

Security Vision wishes you safe deployments and recommends that you always keep a balance between usability and protection. Web tokens are a great helper when used wisely. Now you know everything you want to know about them (and even more) – it's time to put this knowledge into practice. Stay safe!