2024 llama.cpp 7840u: Difference between revisions

From zooid Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
'''Type''': Policy
I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.


'''Status''': Adopted in June 2005
I picked up a Thinkpad P16s with an AMD 7840u to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm.


'''Source File''': http://www.dartmouth.edu/~sustain/about/
Install ROCm, set an env variable for the 780m: <code>export HSA_OVERRIDE_GFX_VERSION=11.0.0</code>


'''Description''':
clone llama.cpp and compile it:


The Dartmouth Sustainability Initiative was created in June of 2005 with the purpose of reducing the environmental impact of the college while integrating sustainability values and practices into the campus culture.
<code>make -j16 LLAMA_HIPBLAS=1 LLAMA_HIP_UMA=1 AMDGPU_TARGETS=gfx1030 DLLAMA_HIP_UMA=
ON</code>
run it like this:
<code>./main -m /home/vid/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M
.gguf -p "example code for a lit Web Component that reverses a string" -n 50 -e
-ngl 33  -n -1</code>


'''Mission'''<br>
ggml_cuda_init: found 1 ROCm devices:
To advance Dartmouth's efforts to become a sustainability leader in higher education by working with senior officers, administrators, faculty, and students to provide the knowledge, skills, and motivation that will integrate sustainability values and practices into the College's strategic planning processes; the management of its resources and operations; facilities planning and design; its research activities; the curriculum; and the extra-curricular life of the College. The Initiative will develop a strategy to embed principles of sustainable prosperity in all of the College's roles - as a place of learning and research, a business enterprise, and a member of the local community.
  Device 0: AMD Radeon Graphics, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.22 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  4095.05 MiB
llm_load_tensors:        CPU buffer size =    70.31 MiB
..............................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch  = 512
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =    64.00 MiB
llama_new_context_with_model: KV self size  =  64.00 MiB, K (f16):  32.00 MiB, V (f16):  32.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =    0.12 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =    81.00 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =    9.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 2
system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
sampling:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 2048, n_predict = -1, n_keep = 1
  simple example code for a lit Web Component that reverses a string
```javascript
import { LitElement, html } from 'lit';
import { customElement, property } from 'lit/decorators.js';
@customElement('reverse-string')
class ReverseString extends LitElement {
  static styles = css`
    :host {
      display: block;
    }
  `;
  @property({ type: String }) input = '';
  render() {
    return html`
      <input type="text" value=${this.input} @input=${this._handleInput} />
      <button @click=${this._reverse}>Reverse</button>
      < p>${this._reversed}< /p>
    `;
  }


'''What is Dartmouth Doing?'''<br>
  _handleInput(event) {
Dartmouth Sustainability is involved in a number of projects promoting Sustainability campus-wide. Here are Examples of efforts being made by us and many other people at Dartmouth.
    this.input = event.target.value;
  }


'''''Sustainable Dining'''''<br>
  _reverse() {
Dining facilities are among the most strategic locations where Dartmouth can demonstrate its commitment to sustainability. Typically, these operations process high throughputs of packaging, energy, and biological resources, generate copious amounts of solid and liquid wastes, and involve large infrastructures. In transitioning toward sustainable dining, a logical first step is to eliminate waste, which is of no-use to individuals or the institution, and is financially and ecologically expensive.
    this._reversed = this.input.split('').reverse().join('');
  }


But here is the good news. You can join the [http://www.dartmouth.edu/~sustain/dartmouth/diningclub.html Sustainable Dining Club] and easily reduce your impact. Also, as one dining hall on campus is already proving, the shift to zero-waste is relatively easy and cost-effective.
  @property private _reversed = '';
}
```
```css
:host {
  display: block;
}
```
This is a simple example of a Lit Web Component that reverses a string. The component has an input field for the user to enter a string, and a button to reverse the string when clicked. The reversed string is displayed below the button.
The component uses the Lit library to define the custom element, and uses the `@customElement` decorator to define the element's name as 'reverse-string'. The `@property` decorator is used to define the input property, and the `static styles` property is used to define the component's styles.
In the `render` method, the input field and button are created using template literals, and the reversed string is displayed using a reactive property `_reversed`.
The input field's value is updated in the `_handleInput` method when the user types in the field, and the string is reversed in the `_reverse` method when the button is clicked. The reversed string is then assigned to the `_reversed` property, which updates the displayed string. [end of text]
 
llama_print_timings:       load time =    2488.76 ms
llama_print_timings:      sample time =      24.92 ms /   485 runs  (    0.05 ms per token, 19462.28 tokens per second)
llama_print_timings: prompt eval time =    576.45 ms /   14 tokens (  41.18 ms per token,    24.29 tokens per second)
llama_print_timings:        eval time =  38985.11 ms /   484 runs  (  80.55 ms per token,    12.41 tokens per second)
llama_print_timings:      total time =  39890.17 ms /  498 tokens
Log end


'''''Energy Reduction and Diversification Measures'''''<br>
With Dartmouth's heating oil bill growing from $4 million to $7 million dollars in one year and the crude oil price setting new record highs topping $72 per barrel, reducing energy use and diversifying Dartmouth's energy portfolio is a priority.


The good news is that the Dartmouth community rose to the occasion in 2006 to reduce energy use by 5.5 percent (January through March) as compared to 2005, even after adjusting for the warmer year. This represents a savings of $210,000. The reductions were a result of campus-wide conservation awareness, lowering building temperatures, student dorm energy challenge and FO&M’s diligence in turning back temperatures when buildings are not occupied.


The Sustainability Initiative is working with Facilities, Operations & Maintenance (FO&M) and Planning Design and Construction (PDC) on energy conservation and high performance buildings and retrofits as well as assessing the feasibility of integrating large scale solar systems into new and/or existing buildings. We are working with FO&M on another project involving updating the campus-wide energy policy on reduction targets for conservation efforts, emissions and renewable energy, and institutional decision making.
{{Blikied|September 27, 2014}}
It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.


The Dartmouth Sustainability Initiative is also working with FO&M on a phased energy conservation awareness campaign. Some of the parts of the campaign include:
{{Blikied|April 13, 2024}}
* Reinvigorating the light switch stickers with new ones that include the Sustainable Dartmouth logo and website where more conservation information can be found.
* Posters in every campus building that are updated monthly and provide feedback on the amount of energy, water and steam used for that building.
 
'''''Indicators'''''<br>
The Dartmouth Sustainability Initiative prepares an annual Dartmouth College Sustainability Report. The report provides a snapshot of campus environmental performance with data sets from as far back as 1995. This report is a voluntary transparent effort to better understand the level of our ecological impact.
 
'''''High Performance Building Design and Construction'''''<br>
The Office of Planning Design and Construction has begun to establish procedures aimed at optimizing the architectural and building system design. Relevant issues include operational and life-cycle costing, efficient technologies, mechanical and electrical equipment, and day lighting.
 
'''''Solid Waste Reduction'''''<br>
With un-recycled wastes at Dartmouth nearing two pounds per person per day, efforts are underway to redirect this trend.
 
Several project include<br>
# Waste-free Dining in Homeplate and Collis
# Compost and recycling in Food court
# Recycle Bins in every dorm room
# Enhanced recycling in the Greek houses
# Compost facility
# Sustainable move-out and move-in
 
''''' Carbon Reduction'''''<br>
The issue of global climate change is likely to be one of the most profound challenges yet faced by humanity.  More poignant is that the horizon to seriously address the catastrophic risks lies during our tenure as leaders.  As the body of science expands the evidence grows more solid that individuals and institutions must reduce their CO<sub>2</sub> emissions to 80 percent below 1990 levels by 2050 and ultimately to within levels that the biosphere has capacity to sequester.
 
'''''Sustainability Coordinators'''''<br>
he Dartmouth Sustainability Initiative (DSI) has trained 48 individuals to be sustainability coordinators in offices, Greek houses and dorms. By engaging the Dartmouth community’s creativity and knowledge of what would work best in their residences or offices, individuals are more likely to take ownership for moving toward sustainability practices. To this end, we have conducted three Sustainability Coordinator Training Retreats to date titled: The Art of Sustainability.
 
This program is designed to train individuals with the skills needed to identify and coordinate practical sustainability projects. These coordinators then solicit ideas from their colleagues to enhance: waste reduction, recycling, furniture re-use, energy reduction, sustainable lunch-rooms, car-pooling, composting, indoor air quality and paper reduction.  Each Coordinator receives a Training Manual that includes:
 
* The steps involved in leading a retreat
* Copies of a sustainability brochure
* A Sustainability Survey for use within their community
* An Eco Audit to assess current practices
* Practical tips and resources
 
'''''Outreach and Communications'''''<br>
The Dartmouth Sustainability Initiative employs several approaches to raise awareness and literacy regarding sustainability and to outreach to the Dartmouth Community and beyond:
 
1. Branding – developed the smooth flowing "S" as a logo which now appears on all campus messaging.<br>
2. Website<br>
3. Campus messaging through:<br>
:1. 5,000 new light-switch stickers with new logo that advertising the website.
:2. Energy Feedback posters in over 50 buildings that report electricity, heat and water as compared to last year and compared to the average building use on campus.  Includes reduction tips.
:3. Social Marketing Campaign in Homeplate
4. Print media interviews<br>
5. Writing Articles<br>
6. Publishing<br>
7. Speaking in classes and at conferences<br>
8. Radio and television interviews
 
[[Category:Colleges]]
[[Category:New Hampshire]]
[[Category:Policies]]
[[Category:Sustainability Policies]]

Revision as of 16:34, 13 April 2024

I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.

I picked up a Thinkpad P16s with an AMD 7840u to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm.

Install ROCm, set an env variable for the 780m: export HSA_OVERRIDE_GFX_VERSION=11.0.0

clone llama.cpp and compile it:

make -j16 LLAMA_HIPBLAS=1 LLAMA_HIP_UMA=1 AMDGPU_TARGETS=gfx1030 DLLAMA_HIP_UMA= ON

run it like this:

./main -m /home/vid/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M .gguf -p "example code for a lit Web Component that reverses a string" -n 50 -e -ngl 33 -n -1

ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.22 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  4095.05 MiB
llm_load_tensors:        CPU buffer size =    70.31 MiB
..............................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =    64.00 MiB
llama_new_context_with_model: KV self size  =   64.00 MiB, K (f16):   32.00 MiB, V (f16):   32.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     0.12 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =    81.00 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =     9.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 2

system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
sampling:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 2048, n_predict = -1, n_keep = 1


 simple example code for a lit Web Component that reverses a string

```javascript
import { LitElement, html } from 'lit';
import { customElement, property } from 'lit/decorators.js';

@customElement('reverse-string')
class ReverseString extends LitElement {

  static styles = css`
    :host {
      display: block;
    }
  `;

  @property({ type: String }) input = ;

  render() {
    return html`
      <input type="text" value=${this.input} @input=${this._handleInput} />
      <button @click=${this._reverse}>Reverse</button>
      < p>${this._reversed}< /p>
    `;
  }
  _handleInput(event) {
    this.input = event.target.value;
  }
  _reverse() {
    this._reversed = this.input.split().reverse().join();
  }
  @property private _reversed = ;
}
```

```css
:host {
  display: block;
}
```

This is a simple example of a Lit Web Component that reverses a string. The component has an input field for the user to enter a string, and a button to reverse the string when clicked. The reversed string is displayed below the button.

The component uses the Lit library to define the custom element, and uses the `@customElement` decorator to define the element's name as 'reverse-string'. The `@property` decorator is used to define the input property, and the `static styles` property is used to define the component's styles.

In the `render` method, the input field and button are created using template literals, and the reversed string is displayed using a reactive property `_reversed`.

The input field's value is updated in the `_handleInput` method when the user types in the field, and the string is reversed in the `_reverse` method when the button is clicked. The reversed string is then assigned to the `_reversed` property, which updates the displayed string. [end of text]
 
llama_print_timings:        load time =    2488.76 ms
llama_print_timings:      sample time =      24.92 ms /   485 runs   (    0.05 ms per token, 19462.28 tokens per second)
llama_print_timings: prompt eval time =     576.45 ms /    14 tokens (   41.18 ms per token,    24.29 tokens per second)
llama_print_timings:        eval time =   38985.11 ms /   484 runs   (   80.55 ms per token,    12.41 tokens per second)
llama_print_timings:       total time =   39890.17 ms /   498 tokens
Log end


Type: Policy

Status: Adopted on 2/15/08

Source File: http://www.usmd.edu/newsroom/news/439

Description:

University System of Maryland (USM) Chancellor William E. Kirwan has launched a new initiative to promote environmental stewardship and sustainable practices across the system's universities, research institutions, and regional higher education centers. The chancellor's Environmental Sustainability and Climate Change Initiative will focus on developing policies, practices, and programs that will make the university system a national leader in institutional responses to the challenges of global climate change.

"Our universities have long been on the forefront of environmental education, research, and policymaking," said Chancellor Kirwan. "The initiative will utilize USM's strengths in these areas to establish sustainable and energy efficient practices across all of our campuses. We will also work to strengthen curricular offerings and opportunities for research and collaboration in environmental science, policy, and sustainable technologies."

"The University System of Maryland recognizes that we need to foster a sustainable environment to retain Maryland's quality of life and leading role in the nation's knowledge economy," said Donald Boesch, president of the University of Maryland Center for Environmental Science, whom Chancellor Kirwan has appointed the system's vice chancellor of environmental sustainability. "The best way to ensure that future generations of Marylanders inherit a healthy, sustainable environment is to lead by example and see to it that today's decisions contribute to a better tomorrow."

Goals of the USM sustainability initiative include:

  • Reducing energy consumption system-wide by 15 percent and greenhouse gas emissions to 1990 levels by the year 2020 per Governor Martin O'Malley's EmPOWER Maryland initiative and Climate Change Commission recommendations as a way to save taxpayers money, reduce stress on Maryland's energy markets, and improve the environment;
  • Crafting and enacting new policies for capital projects, human resources, and procurement to achieve these objectives;
  • Conducting audits of greenhouse gas emission for all USM institutions and using best practices to reduce these emissions;
  • Developing a system-wide strategy for campus sustainability and energy efficiency, including "green" building guidelines and sustainability benchmarks for all new construction and major facility renovations;
  • Refocusing and strengthening instructional and research programs dealing with environmental sustainability and climate change;
  • Bringing the expertise of USM's institutions to bear in addressing Maryland's three-part challenge of Chesapeake Bay restoration, growth, and climate change.

"Across the university system, I have met and spoken with many students who want sustainability to become a guiding principle for our universities and our society," said USM Student Regent Richard Scott, a senior at Frostburg State University and a member of the Maryland Student Climate Coalition (MSCC). The MSCC is a grassroots organization of USM students advocating for the adoption of sustainable practices and a policy of carbon neutrality across the university system. "I am very encouraged by the system's proactive stance and commitment to become a leader in responding to the challenges of global climate change."

The sustainability initiative is one of three new initiatives established by Chancellor Kirwan to address major challenges to Maryland's economic leadership, educational preparedness, and environmental quality. In addition to fostering system-wide sustainability, the chancellor has set two other priorities for his tenure: closing the college retention and graduation achievement gap for minority and underserved students and fostering the state's global economic competitiveness by increasing the number of STEM (science, technology, engineering, and math) teachers and graduates from USM institutions.

For more information on the USM sustainability initiative, visit: http://www.usmd.edu/usm/sustainability. It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.

Type: Policy

Status: Adopted on 2/15/08

Source File: http://www.usmd.edu/newsroom/news/439

Description:

University System of Maryland (USM) Chancellor William E. Kirwan has launched a new initiative to promote environmental stewardship and sustainable practices across the system's universities, research institutions, and regional higher education centers. The chancellor's Environmental Sustainability and Climate Change Initiative will focus on developing policies, practices, and programs that will make the university system a national leader in institutional responses to the challenges of global climate change.

"Our universities have long been on the forefront of environmental education, research, and policymaking," said Chancellor Kirwan. "The initiative will utilize USM's strengths in these areas to establish sustainable and energy efficient practices across all of our campuses. We will also work to strengthen curricular offerings and opportunities for research and collaboration in environmental science, policy, and sustainable technologies."

"The University System of Maryland recognizes that we need to foster a sustainable environment to retain Maryland's quality of life and leading role in the nation's knowledge economy," said Donald Boesch, president of the University of Maryland Center for Environmental Science, whom Chancellor Kirwan has appointed the system's vice chancellor of environmental sustainability. "The best way to ensure that future generations of Marylanders inherit a healthy, sustainable environment is to lead by example and see to it that today's decisions contribute to a better tomorrow."

Goals of the USM sustainability initiative include:

  • Reducing energy consumption system-wide by 15 percent and greenhouse gas emissions to 1990 levels by the year 2020 per Governor Martin O'Malley's EmPOWER Maryland initiative and Climate Change Commission recommendations as a way to save taxpayers money, reduce stress on Maryland's energy markets, and improve the environment;
  • Crafting and enacting new policies for capital projects, human resources, and procurement to achieve these objectives;
  • Conducting audits of greenhouse gas emission for all USM institutions and using best practices to reduce these emissions;
  • Developing a system-wide strategy for campus sustainability and energy efficiency, including "green" building guidelines and sustainability benchmarks for all new construction and major facility renovations;
  • Refocusing and strengthening instructional and research programs dealing with environmental sustainability and climate change;
  • Bringing the expertise of USM's institutions to bear in addressing Maryland's three-part challenge of Chesapeake Bay restoration, growth, and climate change.

"Across the university system, I have met and spoken with many students who want sustainability to become a guiding principle for our universities and our society," said USM Student Regent Richard Scott, a senior at Frostburg State University and a member of the Maryland Student Climate Coalition (MSCC). The MSCC is a grassroots organization of USM students advocating for the adoption of sustainable practices and a policy of carbon neutrality across the university system. "I am very encouraged by the system's proactive stance and commitment to become a leader in responding to the challenges of global climate change."

The sustainability initiative is one of three new initiatives established by Chancellor Kirwan to address major challenges to Maryland's economic leadership, educational preparedness, and environmental quality. In addition to fostering system-wide sustainability, the chancellor has set two other priorities for his tenure: closing the college retention and graduation achievement gap for minority and underserved students and fostering the state's global economic competitiveness by increasing the number of STEM (science, technology, engineering, and math) teachers and graduates from USM institutions.

For more information on the USM sustainability initiative, visit: http://www.usmd.edu/usm/sustainability.