zooid Wiki

2024 llama.cpp 7840u: Difference between revisions

Page Discussion

2024 llama.cpp 7840u (view source)

Revision as of 16:20, 13 April 2024

3 bytes removed ,  13 April 2024
no edit summary
No edit summary
No edit summary
Line 1: Line 1:
I briefly had a Macbook M3 Max with 64GB. It was pretty good at running local LLMs, but couldn't stand the ergonomics and not being able to run Linux, so returned it.
'''Type''': Policy


I picked up a Thinkpad P16s with an AMD 7840u to give Linux hardware a chance to catch up with Apple silicon. It's an amazing computer for the price, and can run LLMs. Here's how I set up llama.cpp to use ROCm.
'''Status''': Adopted in June 2005


Install ROCm, set an env variable for the 780m: <code>export HSA_OVERRIDE_GFX_VERSION=11.0.0</code>
'''Source File''': http://www.dartmouth.edu/~sustain/about/


clone llama.cpp and compile it:
'''Description''':


<code>make -j16 LLAMA_HIPBLAS=1 LLAMA_HIP_UMA=1 AMDGPU_TARGETS=gfx1030 DLLAMA_HIP_UMA=
The Dartmouth Sustainability Initiative was created in June of 2005 with the purpose of reducing the environmental impact of the college while integrating sustainability values and practices into the campus culture.
ON</code>


run it like this:
'''Mission'''<br>
To advance Dartmouth's efforts to become a sustainability leader in higher education by working with senior officers, administrators, faculty, and students to provide the knowledge, skills, and motivation that will integrate sustainability values and practices into the College's strategic planning processes; the management of its resources and operations; facilities planning and design; its research activities; the curriculum; and the extra-curricular life of the College. The Initiative will develop a strategy to embed principles of sustainable prosperity in all of the College's roles - as a place of learning and research, a business enterprise, and a member of the local community.


<code>./main -m /home/vid/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M
'''What is Dartmouth Doing?'''<br>
.gguf -p "example code for a lit Web Component that reverses a string" -n 50 -e
Dartmouth Sustainability is involved in a number of projects promoting Sustainability campus-wide. Here are Examples of efforts being made by us and many other people at Dartmouth.
-ngl 16 -n -1</code>


'''''Sustainable Dining'''''<br>
Dining facilities are among the most strategic locations where Dartmouth can demonstrate its commitment to sustainability. Typically, these operations process high throughputs of packaging, energy, and biological resources, generate copious amounts of solid and liquid wastes, and involve large infrastructures. In transitioning toward sustainable dining, a logical first step is to eliminate waste, which is of no-use to individuals or the institution, and is financially and ecologically expensive.


ggml_cuda_init: found 1 ROCm devices:
But here is the good news. You can join the [http://www.dartmouth.edu/~sustain/dartmouth/diningclub.html Sustainable Dining Club] and easily reduce your impact. Also, as one dining hall on campus is already proving, the shift to zero-waste is relatively easy and cost-effective.
  Device 0: AMD Radeon Graphics, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.22 MiB
llm_load_tensors: offloading 24 repeating layers to GPU
llm_load_tensors: offloaded 24/33 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  2978.91 MiB
llm_load_tensors:        CPU buffer size =  4165.37 MiB
...............................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch  = 512
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =    48.00 MiB
llama_kv_cache_init:  ROCm_Host KV buffer size =    16.00 MiB
llama_new_context_with_model: KV self size  =  64.00 MiB, K (f16):  32.00 MiB, V (f16):  32.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =    0.12 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =  173.04 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =    9.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 92
system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
sampling:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 2048, n_predict = -1, n_keep = 1
  simple example code for a lit Web Component that reverses a string input
Hi there! Here's a simple example of a Lit Web Component that reverses a string input:
```javascript
import { Component, html, css } from 'lit';
class ReverseString extends Component {
  static styles = css`
    input {
      padding: 10px;
      margin-bottom: 20px;
    }
  `;
  static properties = {
    value: { type: String }
  };
  constructor() {
    super();
    this.value = '';
  }
 
  render() {
    return html`
      <style>${this.constructor.styles}</style>
      <input @input=${this._handleInputChange} value=${this.value} type="text">
      < p>Reversed string: ${this._reverseString(this.value)}< /p>
    `;
  }
 
  _reverseString(str) {
    return str.split('').reverse().join('');
  }
  _handleInputChange(event) {
    this.value = event.target.value;
  }
}
customElements.define('reverse-string', ReverseString);
```
In this example, we define a custom Web Component called `reverse-string` that uses Lit for rendering and handling user input. The `ReverseString` class defines a render method that returns an HTML template with an input field and a paragraph that displays the reversed string. The component also defines a `_reverseString` method that reverses a given string using the `split`, `reverse`, and `join` array methods, and a `_handleInputChange` method that updates the component's value whenever the input changes. Finally, we use the `customElements.define` method to register our component with the browser.
You can use this component in your HTML like this:
```html
<reverse-string></reverse-string>
``` [end of text]
llama_print_timings:        load time =    2865.16 ms
llama_print_timings:      sample time =      13.64 ms /  442 runs  (    0.03 ms per token, 32407.07 tokens per second)
llama_print_timings: prompt eval time =    1281.98 ms /    14 tokens (  91.57 ms per token,    10.92 tokens per second)
llama_print_timings:        eval time =  100829.12 ms /  441 runs  (  228.64 ms per token,    4.37 tokens per second)
llama_print_timings:      total time =  102397.06 ms /  455 tokens
Log end


It's definitely not going to win any speed prizes even though is a smaller model, but it could be ok for non time sensitive results, or where using a tiny, faster model is useful.
'''''Energy Reduction and Diversification Measures'''''<br>
With Dartmouth's heating oil bill growing from $4 million to $7 million dollars in one year and the crude oil price setting new record highs topping $72 per barrel, reducing energy use and diversifying Dartmouth's energy portfolio is a priority.


{{Blikied|April 13 12, 2024}}
The good news is that the Dartmouth community rose to the occasion in 2006 to reduce energy use by 5.5 percent (January through March) as compared to 2005, even after adjusting for the warmer year. This represents a savings of $210,000. The reductions were a result of campus-wide conservation awareness, lowering building temperatures, student dorm energy challenge and FO&M’s diligence in turning back temperatures when buildings are not occupied.
 
The Sustainability Initiative is working with Facilities, Operations & Maintenance (FO&M) and Planning Design and Construction (PDC) on energy conservation and high performance buildings and retrofits as well as assessing the feasibility of integrating large scale solar systems into new and/or existing buildings. We are working with FO&M on another project involving updating the campus-wide energy policy on reduction targets for conservation efforts, emissions and renewable energy, and institutional decision making.
 
The Dartmouth Sustainability Initiative is also working with FO&M on a phased energy conservation awareness campaign. Some of the parts of the campaign include:
* Reinvigorating the light switch stickers with new ones that include the Sustainable Dartmouth logo and website where more conservation information can be found.
* Posters in every campus building that are updated monthly and provide feedback on the amount of energy, water and steam used for that building.
 
'''''Indicators'''''<br>
The Dartmouth Sustainability Initiative prepares an annual Dartmouth College Sustainability Report. The report provides a snapshot of campus environmental performance with data sets from as far back as 1995. This report is a voluntary transparent effort to better understand the level of our ecological impact.
 
'''''High Performance Building Design and Construction'''''<br>
The Office of Planning Design and Construction has begun to establish procedures aimed at optimizing the architectural and building system design. Relevant issues include operational and life-cycle costing, efficient technologies, mechanical and electrical equipment, and day lighting.
 
'''''Solid Waste Reduction'''''<br>
With un-recycled wastes at Dartmouth nearing two pounds per person per day, efforts are underway to redirect this trend.
 
Several project include<br>
# Waste-free Dining in Homeplate and Collis
# Compost and recycling in Food court
# Recycle Bins in every dorm room
# Enhanced recycling in the Greek houses
# Compost facility
# Sustainable move-out and move-in
 
''''' Carbon Reduction'''''<br>
The issue of global climate change is likely to be one of the most profound challenges yet faced by humanity.  More poignant is that the horizon to seriously address the catastrophic risks lies during our tenure as leaders.  As the body of science expands the evidence grows more solid that individuals and institutions must reduce their CO<sub>2</sub> emissions to 80 percent below 1990 levels by 2050 and ultimately to within levels that the biosphere has capacity to sequester.
 
'''''Sustainability Coordinators'''''<br>
he Dartmouth Sustainability Initiative (DSI) has trained 48 individuals to be sustainability coordinators in offices, Greek houses and dorms. By engaging the Dartmouth community’s creativity and knowledge of what would work best in their residences or offices, individuals are more likely to take ownership for moving toward sustainability practices. To this end, we have conducted three Sustainability Coordinator Training Retreats to date titled: The Art of Sustainability.
 
This program is designed to train individuals with the skills needed to identify and coordinate practical sustainability projects. These coordinators then solicit ideas from their colleagues to enhance: waste reduction, recycling, furniture re-use, energy reduction, sustainable lunch-rooms, car-pooling, composting, indoor air quality and paper reduction.  Each Coordinator receives a Training Manual that includes:
 
* The steps involved in leading a retreat
* Copies of a sustainability brochure
* A Sustainability Survey for use within their community
* An Eco Audit to assess current practices
* Practical tips and resources
 
'''''Outreach and Communications'''''<br>
The Dartmouth Sustainability Initiative employs several approaches to raise awareness and literacy regarding sustainability and to outreach to the Dartmouth Community and beyond:
 
1. Branding – developed the smooth flowing "S" as a logo which now appears on all campus messaging.<br>
2. Website<br>
3. Campus messaging through:<br>
:1. 5,000 new light-switch stickers with new logo that advertising the website.
:2. Energy Feedback posters in over 50 buildings that report electricity, heat and water as compared to last year and compared to the average building use on campus.  Includes reduction tips.
:3. Social Marketing Campaign in Homeplate
4. Print media interviews<br>
5. Writing Articles<br>
6. Publishing<br>
7. Speaking in classes and at conferences<br>
8. Radio and television interviews
 
[[Category:Colleges]]
[[Category:New Hampshire]]
[[Category:Policies]]
[[Category:Sustainability Policies]]
Retrieved from "https://wiki.zooid.org/w/Special:MobileDiff/5616"