What are tokens

Tokens in LLM are basically currency used by LLMs to identify patterns in an input, in order to generate an output, based on what the LLM has learend. For example, if a text like "big brown fox jumps over the lazy dog" is fed to a LLM model, and an an input is given to instruct LLM to complete a sentence, like "big brown fox jumps.., complete the sentence". Then, LLM will use 8 tokens to complete the sentence.

So how are tokens calculated? Each word in the input is considered as one token, as it's recognized as an pattern by the LLM to complete the sentence. For example:
"big brown fox jumps" is considered 4 tokens, to complete the sentence, LLM will use another 4 tokens. Then, it becomes "big brown fox jumps over the lazy dog" 


LLM will register parameters, and they will have the knowledge, but doesn't necessary means they will answer if you ask them, because based on how you ask them, they will give you the answer.

A is mother of B

Then who is A's son. They don't know

but if we ask who is the mother of B, they will answer A.


Dreaming hallcuination

Fact A, like 010101-01-01010. Lets say this is an actual IC.

but if you ask the AI, is 010101-01-0101. they won't know, because AI will only see how this kind of data is structured, and they will answer like IC will be this form, but they don't know if its real or not. So if you ask them to generate an actual IC, they will generate 010101-02-0101, its not real, but they think it is, because they can see the pattern that IC is this format, so they think is correct, so this is dreaming hallucination.

Comments

Post a Comment

Popular posts from this blog

Support for the experimental syntax 'decorators' isn't currently enabled (5:3):

This guide is targeted specifically to people who encountered this error while using MobX with ReactJS Error Now I know, its frustrating when you got this error, no worries, I know exactly what caused the error. Here was my code that caused the error: // store.js import { observable , action } from 'mobx' ; class ProductStore {   @ observable products = [];   @ action   addProduct ( product ) {     this . products . push ( product );   } } const store = new ProductStore (); export default store ; Solutions tried You might see this message in your console: "Add @babel/plugin-proposal-decorators (https://github.com/babel/babel/tree/main/packages/babel-plugin-proposal-decorators) to the 'plugins' section of your Babel config to enable transformation." and you might have added this new dependencies in both .babelrc [source] , babel.config.json [source] , installing the dependencies in dev using npm [source] , installing it as regular dependency [s...
Read more

Running CQL scripts in Cassandra

How to run CQL scripts in Cassandra  There are 2 ways to run a CQL script, the first way would be running the following command: - cqlsh -f {directory} The second way would be using the command within the CQLSH terminal: - source '{directory}' Where does the {directory} points to? The directory points to the root of your server. For example, if you are on linux, the root would be `/`. Therefore, if you placed your scripts under the folder `scripts`, your directory would look like this `/scripts/{cql_script_name}.cql`.  I've been running the command, but it keeps saying `No such file or directory`? If you met this error, please make sure that:      - The file actually exist in the directory. You can check if by typing `ls -l {directory}`. If it doesn't exist, it will show the same error. If you are on docker, and you are unsure of how to mount your CQL file into your container, you can check out this guide.      - The CQL file has the correct exten...
Read more

How to add environment variables into Heroku container

Image
 Method 1: Using .env file  NOTE: Please add your .env file into .gitignore to not make the file visiblein github. Before we start, please make sure you have ran ' heroku scale worker=1 -a (app) ' in your app.  1) Create an .env file and add your environment variable like so: your-environment-variable=123 2) In your code, import load_dotenv from dotenv like so: import os from dotenv import load_dotenv load_dotenv() test-env = os.getenv('your-environment-variable')  print(test-env)  Note: Please make sure you have requirements.txt present by doing 'pip freeze > requirements.txt' 3) Create your dockerfile FROM python:latest WORKDIR / COPY . . RUN pip install -r requirements.txt CMD [ "python", "test.py" ] 4) Run ' heroku container:push worker -a (app name) ' 5) Run ' heroku container:release worker -a (app name) ' 6) Wait for 10-20 seconds... 7) You can test if your app is working by running ' heroku logs -a (app name)...
Read more