2024 llama.cpp 7840u - Revision history

DavidM at 17:34, 14 April 2024

2024-04-14T17:34:27Z

← Older revision		Revision as of 17:34, 14 April 2024
Line 1:		Line 1:
	~~{{#description2:Using a 7840u for llama.cpp}~~

	I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.		I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.

DavidM at 17:26, 14 April 2024

2024-04-14T17:26:38Z

← Older revision		Revision as of 17:26, 14 April 2024
Line 1:		Line 1:
	{{#~~setmainimage~~:~~https://wiki.zooid.org/images/c/c5/Ipectinstat1-sq-trans2~~.~~png\|thumb\|right}~~}		{{#description2:Using a 7840u for llama.cpp}

	I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.		I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.

DavidM at 17:25, 14 April 2024

2024-04-14T17:25:29Z

← Older revision		Revision as of 17:25, 14 April 2024
Line 1:		Line 1:
			{{#setmainimage:https://wiki.zooid.org/images/c/c5/Ipectinstat1-sq-trans2.png\|thumb\|right}}

	I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.		I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.

DavidM at 16:35, 13 April 2024

2024-04-13T16:35:13Z

← Older revision		Revision as of 16:35, 13 April 2024
Line 107:		Line 107:



	~~{{Blikied\|September 27, 2014}}~~
	It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.		It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.

	{{Blikied\|April 13, 2024}}		{{Blikied\|April 13, 2024}}

DavidM at 16:34, 13 April 2024

2024-04-13T16:34:58Z

← Older revision		Revision as of 16:34, 13 April 2024
Line 72:		Line 72:
	`;		`;
	}		}

	_handleInput(event) {		_handleInput(event) {
	this.input = event.target.value;		this.input = event.target.value;
	}		}

	_reverse() {		_reverse() {
	this._reversed = this.input.split('').reverse().join('');		this._reversed = this.input.split('').reverse().join('');
	}		}

	@property private _reversed = '';		@property private _reversed = '';
	}		}

DavidM at 16:34, 13 April 2024

2024-04-13T16:34:15Z

Show changes

DavidM at 16:20, 13 April 2024

2024-04-13T16:20:14Z

← Older revision		Revision as of 16:20, 13 April 2024
Line 110:		Line 110:
	It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.		It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.

	{{Blikied\|April 13 12, 2024}}		{{Blikied\|April 13, 2024}}

DavidM at 14:40, 13 April 2024

2024-04-13T14:40:00Z

← Older revision		Revision as of 14:40, 13 April 2024
Line 1:		Line 1:
	I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.		I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.

	I picked up a Thinkpad P16s with an AMD ~~7840~~ to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm.		I picked up a Thinkpad P16s with an AMD 7840u to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm.

	Install ROCm, set an env variable for the 780m: <code>export HSA_OVERRIDE_GFX_VERSION=11.0.0</code>		Install ROCm, set an env variable for the 780m: <code>export HSA_OVERRIDE_GFX_VERSION=11.0.0</code>
Line 107:		Line 107:
	llama_print_timings: total time = 102397.06 ms / 455 tokens		llama_print_timings: total time = 102397.06 ms / 455 tokens
	Log end		Log end

			It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.

	{{Blikied\|April 13 12, 2024}}		{{Blikied\|April 13 12, 2024}}

DavidM at 14:38, 13 April 2024

2024-04-13T14:38:21Z

← Older revision		Revision as of 14:38, 13 April 2024
Line 17:		Line 17:


			ggml_cuda_init: found 1 ROCm devices:
			Device 0: AMD Radeon Graphics, compute capability 11.0, VMM: no
			llm_load_tensors: ggml ctx size = 0.22 MiB
			llm_load_tensors: offloading 24 repeating layers to GPU
			llm_load_tensors: offloaded 24/33 layers to GPU
			llm_load_tensors: ROCm0 buffer size = 2978.91 MiB
			llm_load_tensors: CPU buffer size = 4165.37 MiB
			...............................................................................................
			llama_new_context_with_model: n_ctx = 512
			llama_new_context_with_model: n_batch = 512
			llama_new_context_with_model: n_ubatch = 512
			llama_new_context_with_model: freq_base = 1000000.0
			llama_new_context_with_model: freq_scale = 1
			llama_kv_cache_init: ROCm0 KV buffer size = 48.00 MiB
			llama_kv_cache_init: ROCm_Host KV buffer size = 16.00 MiB
			llama_new_context_with_model: KV self size = 64.00 MiB, K (f16): 32.00 MiB, V (f16): 32.00 MiB
			llama_new_context_with_model: ROCm_Host output buffer size = 0.12 MiB
			llama_new_context_with_model: ROCm0 compute buffer size = 173.04 MiB
			llama_new_context_with_model: ROCm_Host compute buffer size = 9.01 MiB
			llama_new_context_with_model: graph nodes = 1030
			llama_new_context_with_model: graph splits = 92

			system_info: n_threads = 8 / 16 \| AVX = 1 \| AVX_VNNI = 0 \| AVX2 = 1 \| AVX512 = 1 \| AVX512_VBMI = 1 \| AVX512_VNNI = 1 \| FMA = 1 \| NEON = 0 \| ARM_FMA = 0 \| F16C = 1 \| FP16_VA = 0 \| WASM_SIMD = 0 \| BLAS = 1 \| SSE3 = 1 \| SSSE3 = 1 \| VSX = 0 \| MATMUL_INT8 = 0 \|
			sampling:
			repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
			top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
			mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
			sampling order:
			CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
			generate: n_ctx = 512, n_batch = 2048, n_predict = -1, n_keep = 1


			simple example code for a lit Web Component that reverses a string input

			Hi there! Here's a simple example of a Lit Web Component that reverses a string input:

			```javascript
			import { Component, html, css } from 'lit';

			class ReverseString extends Component {
			static styles = css`
			input {
			padding: 10px;
			margin-bottom: 20px;
			}
			`;

			static properties = {
			value: { type: String }
			};

			constructor() {
			super();
			this.value = '';
			}

			render() {
			return html`
			<style>${this.constructor.styles}</style>
			<input @input=${this._handleInputChange} value=${this.value} type="text">
			< p>Reversed string: ${this._reverseString(this.value)}< /p>
			`;
			}

			_reverseString(str) {
			return str.split('').reverse().join('');
			}

			_handleInputChange(event) {
			this.value = event.target.value;
			}
			}

			customElements.define('reverse-string', ReverseString);
			```

			In this example, we define a custom Web Component called `reverse-string` that uses Lit for rendering and handling user input. The `ReverseString` class defines a render method that returns an HTML template with an input field and a paragraph that displays the reversed string. The component also defines a `_reverseString` method that reverses a given string using the `split`, `reverse`, and `join` array methods, and a `_handleInputChange` method that updates the component's value whenever the input changes. Finally, we use the `customElements.define` method to register our component with the browser.

			You can use this component in your HTML like this:

			```html
			<reverse-string></reverse-string>
			``` [end of text]

			llama_print_timings: load time = 2865.16 ms
			llama_print_timings: sample time = 13.64 ms / 442 runs ( 0.03 ms per token, 32407.07 tokens per second)
			llama_print_timings: prompt eval time = 1281.98 ms / 14 tokens ( 91.57 ms per token, 10.92 tokens per second)
			llama_print_timings: eval time = 100829.12 ms / 441 runs ( 228.64 ms per token, 4.37 tokens per second)
			llama_print_timings: total time = 102397.06 ms / 455 tokens
			Log end

	{{Blikied\|April 13 12, 2024}}		{{Blikied\|April 13 12, 2024}}

DavidM: Created page with "I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it. I picked up a Thinkpad P16s with an AMD 7840 to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm. Install ROCm, set an env variable for the 780m: export HSA_OVERRIDE_GFX_VERSION=11.0.0 2024-04-13T14:23:33Z Created page with "I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it. I picked up a Thinkpad P16s with an AMD 7840 to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm. Install ROCm, set an env variable for the 780m: <code>export HSA_OVERRIDE_GFX_VERSION=11.0.0</co..." New page I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it. I picked up a Thinkpad P16s with an AMD 7840 to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm. Install ROCm, set an env variable for the 780m: <code>export HSA_OVERRIDE_GFX_VERSION=11.0.0</code> clone llama.cpp and compile it: <code>make -j16 LLAMA_HIPBLAS=1 LLAMA_HIP_UMA=1 AMDGPU_TARGETS=gfx1030 DLLAMA_HIP_UMA= ON</code> run it like this: <code>./main -m /home/vid/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M .gguf -p "example code for a lit Web Component that reverses a string" -n 50 -e -ngl 16 -n -1</code> {{Blikied|April 13 12, 2024}}