ArxivLlama/README.md

---
base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
license: apache-2.0
language:
- en
---

# Model usage example

```python
import transformers
import torch

torch.set_default_tensor_type(torch.cuda.HalfTensor)


model_id = "xinyifang/ArxivLlama"

pipeline = transformers.pipeline(
    "text-generation",
    model = model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

def LLMClassifier(description):

    messages = [
        {"role": "system", "content": "You are a classifier that determines which category a given node belongs to from arxiv cs. The 40 categories are: 0-(Numerical Analysis) 1-(Multimedia) 2-(Logic in Computer Science) 3-(Computers and Society) 4-(Cryptography and Security) 5-(Distributed, Parallel, and Cluster Computing) 6-(Human-Computer Interaction) 7-(Computational Engineering, Finance, and Science) 8-(Networking and Internet Architecture) 9-(Computational Complexity)\
          10-(Artificial Intelligence) 11-(Multiagent Systems) 12-(General Literature) 13-(Neural and Evolutionary Computing) 14-(Symbolic Computation) 15-(Hardware Architecture) 16-(Computer Vision and Pattern Recognition) 17-(Graphics) 18-(Emerging Technologies) 19-(Systems and Control)\
          20-(Computational Geometry) 21-(Other Computer Science) 22-(Programming Languages) 23-(Software Engineering) 24-(Machine Learning) 25-(Sound) 26-(Social and Information Networks) 27-(Robotics) 28-(Information Theory) 29-(Performance)\
          30-(Computation and Language) 31-(Information Retrieval) 32-(Mathematical Software) 33-(Formal Languages and Automata Theory) 3 4-(Data Structures and Algorithms) 35-(Operating Systems) 36-(Computer Science and Game Theory) 37-(Databases) 38-(Digital Libraries) 39-(Discrete Mathematics). Output the corresponding number only and nothing else."},
        {"role": "user", "content": f"{description}"},
    ]

    outputs = pipeline(
    messages,
    max_new_tokens=128000,
)
    return outputs[0]["generated_text"][-1]['content']

# Adding more neighbors can help with the accuracy, for example, "- Node Title: '{neighbor_title}' ({hop} hop)\n  Abstract: '{neighbor_abstract}'\n"
description = "Graph representation learning has become a critical task across various domains such as social networks and recommender systems. Recently, the rise of large language models (LLMs) has opened up new possibilities for processing text-attributed graphs (TAGs), where nodes are associated with textual information. Despite promising progress, applying LLMs to TAGs faces significant challenges, including input window size limitations and the computational overhead of handling large-scale graphs. To address these challenges, we propose a novel approach that leverages a multi-profiling framework as a data augmentation method, thereby increasing the diversity and quantity of the training samples. The profiles/summaries generated by each of the five profiling/summarizing models are then combined with the graph structure, prompts and ground-truth labels to create a comprehensive and varied fine-tuning data. By strategically selecting profiling models with an appropriate number of neighboring nodes and constructing concise yet informative fine-tuning prompts, our proposed approach enables LLMs to process more complex graphs while operating within limited computational resources. Notably, our experiments demonstrate that it is unnecessary to construct intricate graph structures for fine-tuning to achieve strong performance. Our approach outperforms 11 state-of-the-art baselines, achieving 74.31\% accuracy and 85.15\% accuracy in two large real-world benchmark datasets, ogbn-arxiv and ogbn-products, respectively. Furthermore, our model is publicly available on Hugging Face"

predict_label = LLMClassifier(description)
print(predict_label) # 30
```

# Uploaded  model

- **Developed by:** xinyifang
- **License:** apache-2.0
- **Finetuned from model :** unsloth/meta-llama-3.1-8b-instruct-bnb-4bit

This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)

# Citation

This model is from our paper [LLM Profiling and Fine-Tuning with Limited Neighbor Information for Node Classification on Text-Attributed Graphs](https://ieeexplore.ieee.org/abstract/document/11401710):

```bibtex
@INPROCEEDINGS{11401710,
  author={Fang, Xinyi and Lee, Kyumin and Li, Yichuan},
  booktitle={2025 IEEE International Conference on Big Data (BigData)}, 
  title={LLM Profiling and Fine-Tuning with Limited Neighbor Information for Node Classification on Text-Attributed Graphs}, 
  year={2025},
  volume={},
  number={},
  pages={5613-5622},
  keywords={Training;Representation learning;Accuracy;Social networking (online);Computational modeling;Large language models;Data models;Tuning;Recommender systems;Faces},
  doi={10.1109/BigData66926.2025.11401710}}
```